During the pandemic crisis, many managers and team leaders are experiencing remote working for the first time, and I’m sure there are a lot of questions and concerns on their minds. Think of me as an agony uncle for your darkest fears on remote workers…
Dear Coding Daddy, My team are a decent lot who really know their stuff. However, now they’re remote, I worry that they will be finding it difficult to track down any inter-team help they need. I’m concerned that they may spend too much time blocked, trying to get in touch with people who they could just wander over to in the office.
Miss De-av Manageur
Dear De-av Manageur,
The two problems here are “finding someone who can help with a thing” and “getting in touch with that person”. In the office, a lack of traceable contacts is often glossed over because people can just stand up and shout “who can help me with X?”. Well this is somewhat still the case in a remote scenario, as long as everyone is in the same timezone and there’s someone in the team who has an answer, but let’s consider this isn’t the case.
A live chat application is a must for remote working. Tools such as Slack or MS Teams are excellent choices. Something I have found helps is having a pinned message at the top of a ‘problems’ or ‘blocked’ channel which has the tags of every first point of contact in each external team. That will allow anyone to click that tag and start talking with someone who can help them, right away. It’s also a good idea to have a backup contact for each team as well.
The important thing to take away from this, is that something as simple as a pinned message can save time by making it easy for people to ask questions. It’s too easy in a remote scenario to try to answer something without help – communication needs to be encouraged and embraced.
Dear Coding Daddy, I have a dozen team members working remotely and I need to run a retrospective. When there’s only one or two people working remote, it’s hard enough to get them engaged – how do I manage now everyone’s remote?
Dr Revu Werk
Dear Dr Werk,
Firstly, don’t panic – remote meetings can be surprisingly straight forward if you take some time to prepare the tech. I can personally recommend Miro and Office 365 – either of these will allow you to share a document and work collaboratively, which is key in a retro. Miro will also provide video chat, and although Office 365 covers this as well, you will have to work out yourself how you want to incorporate the different applications Microsoft provide. You may end up using a combination.
Secondly, have some fun with this. There’s nothing wrong with getting it wrong. The important thing with a retro is to get everyone engaged, so maybe have a social online meet before the retro, to loosen things up a bit. Laugh at the bits that don’t work, and celebrate the fails.
Difficulties integrating a couple of remote workers into meetings are common, but when everyone is remote, things get easier. The tech either works or it doesn’t, but it can still be engaging because everyone is in the same boat. Even if you end up sharing an Excel Online doc and just writing messages, it’s a start and the only way is up.
Dear Coding Daddy, I can’t see my team any more, now they’re all “working from home”! How do I know they’re actually working?!
Mr Pani King
Let’s start with some team management basics. Firstly, hire people you think you can trust. There’s no point driving team members to work when they don’t want to – the work they do will be second rate and shoddy. Focus on the people who are actually willing to work and let the others fall away. Secondly, if you are planning work well, there should be stories delivered constantly; the fact that work moves through the board in one direction (and doesn’t keep coming back) shows that people are working.
So now you are focused on the people worth focusing on, think about how you can enable them to do more work, rather than how you can make them do more work. Generally, if a task can be easily completed in a couple of hours by one or two people, then it’ll be done pretty quickly. Problems arise when the task is going to take several days and involve a dozen different people from different teams, each with their own work to do. If you have this problem, then you aren’t properly controlling the flow of work – you aren’t breaking work down into simple deliverables and you aren’t keeping a check on cross-team dependencies.
For development work, sometimes cross team dependencies can be reduced by making sure other teams have completed their work before your team starts theirs, but it’s more common to find architectural issues at the heart of the problem. Teams built around different tiers, leading to domain logic being split across different systems – too many systems and we hit gridlock. This can be a harder thing to fix, as it involves introducing very different ways of working from what’s probably gone on before. When tackling problems like this – start small and demonstrate the success.
Dear Coding Daddy, Some of my team have terrible internet connections. Others are managing with less than optimal hardware. The company VPN is also showing signs of overload. How can I help my team?
Ms Dia Lupp
Dear Ms Lupp,
If Maslow had considered remote working when writing his hierarchy of needs, he would have included a decent internet connection and proper video/audio equipment. Presumably, if you team members are working remotely but have poor internet, this is in some way forced and needs to be fixed while that individual is working remotely (such as in the COVID-19 pandemic). This means your path forwards is clear: get them a better connection. It’s quite possible these days to have a great experience using mobile data – the business could send a 5/4G data hub for business use. And if there are other hardware problems, send them something better. These one off expenses are vastly preferable to the ongoing expense of sub-optimal working conditions.
There is nothing more important than enabling smooth working conditions for your team. A remote experience can be rewarding, but it can also be isolating and unhealthy if not done correctly.
If you company VPN is struggling and it isn’t going to be upgraded any time soon, try not using it. This will likely mean you can’t use some of the business’ preferred software for collaboration, but getting your team to the point where they can work together is just step 1 – it’s the most basic of needs and can’t be ignored. If you find you need to use external tools which haven’t been approved, make sure to talk to your security team – you may have to police the content exposed in that tool if privacy and confidentiality is an issue.
Ok, so that was mostly a bit of fun with a few obvious tips and hints. Maybe one of the most important things to realise about your remote teams right now is that everyone is stressed about the virus and what limits are being placed on their life. Don’t add to their stress, be a friend, be patient, listen to what they need from you. No-one is sat at home thinking how to get one over on you. Your team wants to work, help them to help you.
With the current concerns about Coronovirus, everyone seems to be talking about the challenges which home working presents, for both workers and managers. Various articles are giving some great advice about how to get round problems; I thought I’d focus more on the advantages that remote working provides.
You’re in control
The most immediately obvious benefit for anyone working remotely is that they have full control over how much contact they have with their team. Yes, teams need to be able to work together, but it’s very normal for a big chunk of time to be spent individually focusing on a task. Even given the growth of pair programming and mob coding, not everyone is a developer and not every developer pairs all the time (or even at all). In an office environment, you will probably have half a dozen people within arm’s reach of you who could be visited at any moment by any number of people wanting to have a discussion. As office workers, we’ve had to develop the ability to ‘turn off’ these nearby conversations but often we are still working in a distracted state, or worse, we end up being pulled into a conversation we really didn’t need to be involved with.
The remote working experience is refreshingly absent of such distractions. It might take a while to tune your Slack or MS Teams alerts so you only see things that are relevant, but most get to a good situation pretty quickly. When you want to focus on something, you can do so without interruptions.
Everyone is available
It might seem surprising, but when everyone is working remotely, people don’t tend to wander away from their desks to have conversations. If you need to contact someone and the are working, they’ll be able to at least ping you a reply saying they’re in the middle of something – you’ll rarely be left hanging.
This is my personal favourite thing about working remotely. Wasting maybe two hours a day driving to and from the office, paying for parking and fuel – who wants to do that? Have an extra hour in bed, get an early start on your day, go for a run, have breakfast with friends – you know, live your life!
You can also pat yourself on the back for not contributing to daily traffic problems and harmful emission levels.
Work / life balance
There are few people more dedicated to their work than I am – I love what I do, I’m lucky. But even I want to be available for my family when they need me, not stuck miles away in an office that I can’t really leave unless I plan to finish for the day. If my son needs to be off school for a day, I don’t want the headache of having to find someone else to look after him – I want to be there. That’s not always possible when there are deadlines and no provision for remote working.
A level field
When working remotely isn’t the norm, people who do spend time out of the office can find themselves at a disadvantage. How often are apologies made for remote workers not being able to see a whiteboard during a meeting? If everyone is remote, these issues are already solved. In fact, because it has to work well, it does. Who doesn’t enjoy playing with the wide and ever growing range of collaboration tools?
There are advantages for businesses as well as the workforce. With remote workers, there isn’t the need for large dedicated office space. Scaling down on office spaces can literally save millions. When there is a need for everyone to be in the same place, there are plenty of shared spaces that can be used, which encourages an entire industry.
There’s something about inviting your work colleagues into your own space (even virtually) rather than meeting in a space which isn’t personal to anyone, which humanises the experience. When it’s normal for someone’s spouse to wander into view of the webcam to bring a coffee in the middle of a meeting, it extends the level of familiarity with each other beyond who we are at work – people see the person they’re talking to as a wife, a parent, an artist, a musician, as so much more than ‘just a Project Manager’.
In general, people feel safer at home. Dealing with conflict is something we all have to get used to but working in your own safe space can help keep perspective. Professional conflict should be kept professional, which is made easier when you’re sat in your own home.
Change of scenery
Working remotely means you could conceivably work from pretty much anywhere in the world. Where do you want to be today? A coffee shop? A different city? A different country?
The future of working?
In the UK, the new high speed train system is meant to bring an extra million people daily into London. Is this really a good idea? Is it going to improve anyone’s life? If humanity is going to find a way to work without putting the planet at risk, moving people on a daily basis is probably not it. The technology to work remotely is already here – we should embrace it and take advantage of it. It’s a better way of living and working for people and businesses alike.
Although I see developers writing more and more tests, their efforts are often ignored by QA and not taken into account by the test strategy. It’s common for developers to involve QA in their work, but this is not the full picture. To maximise on efficiency of test coverage, I think developer tests should be accounted for as part of the test approach.
The role of QA as the gateway of quality, rather than the implementors of quality, allows for good judgement to be used when deciding if a developer’s tests provide suitable coverage to count toward the traditional QA effort. This requires QA to include some development skills, so the role is capable of seeing the benefits and flaws in developer written tests.
What follows is a breakdown of how I categorise different types of test, followed by a “bringing it all together” section where I hope to outline a few approaches for streamlining the amount of testing done.
Unit testing can take the form of a few different flavours. I’ve noticed some difference depending on language and platform. There are also different reasons for writing unit tests.
Post development tests
These are the unit tests developers would write before TDD became popular. They do have a lot of value. They are aimed at ensuring business logic does what it’s supposed to at the time of development, and keeps on working at the time of redevelopment five years later.
I fall into the crowd who bleieve TDD is more about code design than it is about functional correctness. Taking a TDD approach will help a developer write SOLID code, and hopefully make it much easier to debug and read. Having said that, it’s a fantastic tool for writing complex business logic, because generally you will have had a very kind analyst work out exactly what the output of your complex business logic is expected to be. Tackling the dev with a TDD approach will make translating that logic into code much easier, as it’s immediately obvious when you break something you wrote 2 minutes ago.
I’m a big believer in BDD tests being written at the unit level. For me, these are tests which are named and namespaced in a way to indicate the behaviour being tested. They will often have most of the code in a single setup method and test the result of running the setup in individual test methods, named appropriately. These can be used in the process of TDD, to design the code, and they can also be excellent at making a connection between acceptance criteria and business logic. Because the context of the test is important, I find there’s about a 50/50 split of when I can usefully write unit tests in a BDD fashion vs working with a more general TDD approach. I’ve also found that tests like these can encourage the use of domain terminology from the story being worked on, as a result of the wording of the AC’s.
Without a doubt, BDD style unit tests are much better at ensuring important behaviours remain unbroken than more traditional class level unit tests, because the way they’re named and grouped makes the purpose of the tests, and the specific behaviour under test, much clearer. I can tell very quickly if a described behaviour is important or not, but I can’t tell you whether the method GetResult() should always return the number 5. I would encourage this style of unit testing where business logic is being written.
However! BDD is not Gherkin. BDD tests place emphasis on the behaviour being tested rather than just on the correctness of the results. Don’t be tied to arbitrarily writing ‘given, when, then’ statements.
Hitting a database
The DotNet developer in me screams “No!” whenever I think about this, but the half of me which loves Ruby on Rails understands that because MySQL is deployed with a single ‘apt’ command and because I need to ensure my dynamically typed objects save and load correctly, hitting the db is a really good idea. My experience with RoR tells me that abstracting the database is way more work than just installing it (because ‘apt install mysql2’ is far quicker to write than any number of mock behaviours). In the DotNet world, you have strongly typed objects, so cheking that an int can be written to an integer column in a database is a bit redundant.
Yes, absolutely, this is blatantly an integration test. When working with dynamic languages (especially on Linux) I think the dent in the concept is worth the return.
Scope of a unit test
This is important, because if we are going to usefully select tests to apply toward overall coverage, we need to know what is and isn’t being tested. There are different views on unit testing, on what does and doesn’t constitute a move from unit testing to integration testing, but I like to keep things simple. My view is that if a test crosses application processes, then it’s an integration test. If it remains in the application you are writing (or you’re hitting a db in RoR) and doesn’t require the launching of the application prior to test, then it’s a unit test. That means unit tests can cover a single method, a single class, a single assembly, or multiple of all or any of these. Pick whatever definition of ‘a unit’ works to help you write the test you need. Don’t be too constrained on terminology – if you need a group of tests to prove some functionality which involves half a dozen assemblies in your application, write them and call them unit tests. They’ll run after compile just fine. Who cares whether someone else’s idea of a ‘unit’ is hurt?
Who writes these?
I really hope that it’s obvious to you that developers are responsible for writing unit tests. In fact, developers aren’t only responsible for writing unit tests, they’re responsible for realising that they should be writing unit tests and then writing them. A developer NEVER has to ask permission to write a unit test, any more than they need permission to turn up to work. This is a part of their job – does a pilot need to ask before extending the landing gear?
How can QA rely on unit tests?
Firstly, let’s expel the idea that QA might ‘rely’ on a unit test. A unit test lives in the developer’s domain and may change at any moment or be deleted. As is so often the case in software development, the important element is the people themselves. If a QA and a dev work together regularly and the QA knows there are unit tests, has even seen them and understands how the important business logic is being unit tested, then that QA has far greater confidence that the easy stuff is probably OK. Hopefully QA have access to the unit test report from each build, and the tests are named well enough to make some sense. With this scenarion, it’s easier to be confident that the code has been written to a standard that is ready for a QA to start exposing it to real exploratory “what if” testing, rather than just checking it meets the acceptance criteria. Reasons for story rejection are far less likely to be simple logic problems.
I might upset some hardware people, but in my mind a component test is ran against your code when it is running, but without crossing application boundaries downstream. So if you have written a DotNet Web API service, you would be testing the running endpoint of that service while intercepting downstream requests and stubbing the responses. I’ve found Mountebank to be an excellent tool for this, but I believe there are many more to choose from.
TDD and BDD
Component tests can be ran on a developer’s machine, so it’s quite possible for these to be useful in a TDD/BDD fashion. The downside is that the application needs to be running in order to run the tests, so if they are to be executed from a build server, then the application needs to be started and the stubbing framework set up – this can be trickier to orchestrate. As with any automation, this is only really tricky the first time it’s done. After that, the code and patterns are in place.
In my experience, component tests have limited value. I’ve found that this level of testing is often swallowed by the combination of good unit testing and good integration testing. From a point of view, they are actually integration tests, as you are testing the integration between the application and the operating system.
Having said that, if the downstream systems are not available or relilable, then this approach allows the application functionality to be tested seperately from any integrations.
Who writes these?
Developers write component tests. They may find that they are making changes to an older application, which thiey don’t fully understand. Being able to sandbox the running app and stub all external dependencies can help in these situations.
How can QA rely on component tests?
This again comes down to the very human relationship between the QA and the developer. If there is a close working relationship, and the developer takes the time to show the QA the application under test and explain why they’ve tested it in that way, then it increases the confidence the QA has that the code is ready for them to really go to town on it. It might be that in a discussion prior to development, the QA had suggested that they would be comfier if this component test existed. The test could convey as much meaning as the AC’s in the story.
Again, quality is ensured by having a good relationship between the people involved in building it.
Application Scoped Integration Tests
Integration tests prove that the application functions correctly when exposed to the other systems it will be talking to once in production. They rely on the application being installed and running, and they rely on other applications in ‘the stack’ also running. Strictly speaking, any test that crosses application boundaries is an integration test, but I want to focus on automated tests. We haven’t quite got to manual testing yet.
TDD and BDD
With an integration test, we are extending the feedback time a little too far for it to be your primary TDD strategy; you may find it useful to write an integration test to show a positive and negative result from some business logic at the integration level before that logic is deployed just so you can see the switch from failing to passing tests, but you probably shouldn’t be testing all boundary results at an integration level as a developer. It’s definitely possible to write behaviour focussed integration tests. If you’re building an API and the acceptance criteria includes a truth table, you pretty much have your behvaviour tests already set out for you, but consider what you are testing at the integration level – if you have unit tests proving the logic, then you only need to test that the business logic is hooked in correctly.
The difficult part of this kind of testing often seems to be setting up the data in downstream systems for the automated tests. I find it difficult to understand why anyone would design a system where test data can’t be injected at will – this seems an obvious part of testable architecture; a non-functional requirement that keeps getting ignored for no good reason. If you have a service from which you need to retrieve a price list, that service should almost certainly be capable of saving price list data sent to it in an appropriate way; allowing you to inject your test data.
The title of this section is ‘Application Scoped Integration Tests’ – my intention with that title is to draw a distinction between tests which are intended to test the entire stack and tests which are intended to test the application or service you are writing at the time. If you have 10 downstream dependencies in your architecture, these tests would hit real instances of these dependencies but you are doing this to test the one thing you are building (even though you will generally catch errors from further down the stack as well).
Who writes these?
These tests are still very closely tied with the evolution of an application as it’s being built, so I advocate for developers to write these tests.
How can QA rely on application scoped integration tests?
Unit and component tests are written specifically to tell the developer that their code works; integration tests are higher up the test pyramid. This means they are more expensive and their purpose should be considered carefully. Although I would expect a developer to write these tests alongside the application they are building, I would expect significant input from a QA to help with what behaviours should and shouldn’t be tested at this level. So we again find that QA and dev working closely gives the best results.
Let’s consider a set of behaviours defined in a truth table which has different outcomes for different values of an enum retrieved from a downstream system. The application doesn’t have control over the vaules in the enum; it’s the downstream dependency that is returning them, so they could conceivably change without the application development team knowing it. At the unit test level, we can write a test to prove every outcome of the truth table. At the integration level, we don’t need to re-write those tests, but we do need to verify that the enum contains exactly the values we are expecting, and what happens if the enum can’t be retrieved at all.
Arriving at this approach through discussion between QA and Dev allows everyone to understand how the different tests at different levels compliment each other to prove the overall functionality.
These are probably my favourite type of test! Generally written for APIs and message processors, these tests can prevent regression issues caused by services getting updated beyond the expectation of consumers. When a developer is writing something which must consume a service (whether via synchronous or asynchronous means) they write tests which execute against the downstream service to prove it behaves in a way that the consumer can handle.
For example: if the consumer expects a 400 HTTP code back when it POSTs an address with no ‘line 1’ field, then the test will intentionally POST the invalid address and assert that the resulting HTTP code is 400. This gets tested because subsequent logic in the consumer relies on having received the 400; if the consumer didn’t care about the response code then this particular consumer contract wouldn’t include this test.
The clever thing about these tests is when they are ran: the tests are given to the team who develop the consumed service and are ran as part of their CI process. They may have similar tests from a dozen different consumers, some testing the same thing, the value is that it’s immediately obvious to the service developers who relies on what behaviour. If they break something then they know who will be impacted.
This is the subject of some disagreement. There is a school of thought which suggests nothing more than the shape of the request, the shape of the response, and how to call the service should be tested; beyond that should be a black box. Personally, I think that while this is probably correct most of the time, there should be some wiggle room, depending on how coupled the services are. If a service sends emails, then you might want to check for an ’email sent’ event being raised after getting a successful HTTP response (even though the concept of raising the event belongs to the emailer service) – the line is narrow, testing too deep increases coupling but all situations are different.
Who writes these?
These are written by developers and executed in CI.
How can QA rely on consumer contracts?
Consumer contracts are one of the most important classes of tests. If the intention is ever to achieve continuous delivery, these types of test will become your absolute proof that a change to a service hasn’t broken a consumer. Because they test services, not UI, they can be automated and all encompassing but they might not test at a level that QA would normally think about. To get a QA to understand these tests, you will probably have to show how they work and what they prove. They will execute well before any code gets to a ‘testing’ phase, so it’s important for QA to understand the resilience that the practice brings to distributed applications.
Yet again we are talking about good communication between dev and QA as being key to proving the tests are worth taking into consideration.
Stack Scoped Integration Tests
These are subtley different from application scoped integration tests.
There are probably respected technologists out there who will argue that these are the same class of test. I have seen them combined and I have seen them adopted individually (or more often not adopted at all) – I draw a destinction because of the different intentions behind writing them.
These tests are aimed at showing the interactions between the entire stack are correct from the point of view of the main entry point. For example, a microservice may call another service which in turn calls a database. A stack scoped test would interact with the first microservice in an integrated environment and confirm that relevant scenarios from right down the stack are handled correctly.
TDD and BDD
You would be forgiven for wondering how tests at such a high level can be included in TDD or BDD efforts; the feedback loop is pretty long. I find these kind of tests are excellent for establishing behaviours around NFRs, which are known up front so failing tests can be put in place. These are also great at showing some happy path scenarios, while trying to avoid full on boundary testing (simply because the detail of boundary values can be tested much more efficiently at a unit level). It might be worth looking at the concept of executable specifications and tools such as Fitnesse – these allow behaviours to be defined in a heirarchical wiki and linked directly to both integration and unit tests to prove they have been fulfilled. It’s an incredibly efficient way to produce documentation, automated tests, and functioning code at the same time.
Being scoped to the stack means that there is an implicit intention for these tests to be applied beyond the one service or application. We are expecting to prove integrations right down the stack. This also means that it might not be just a developer writing these. If we have a single suite of stack tests, then anyone making changes to anything in the stack could be writing tests in this suite. For new features, it would also be efficient if QA were writing some of these tests; this can help drive personal integration between dev and QA, and help the latter get exposure to what level of testing has already been applied before it gets anywhere near a classic testing phase.
These tests can be brittle and expensive if the approach is wrong. Testing boundary values of business logic at a stack level is inefficient. That isn’t to say that you can’t have an executable specification for your business logic, just that it possibly shouldn’t be set up as an integration test – perhaps the logic could be tested at multiple levels from the same suite.
How can QA rely on stack scoped integration tests?
These tests are quite often written by a QA cooperating closely with a BA, especially if you are using something like Fitnesse and writing executable specifications. A developer may write some plumbing to get the tests to execute against their code in the best way. Because there is so much involvement from QA, it shouldn’t be difficult for these tests to be trusted.
I think this type of testing applied correctly demonstrates the pinacle of cooperation between BA, QA, and dev; it should always result in great product.
Automated UI Tests
Many user interfaces are browser based, and as such need to be tested in various different browsers and different versions of each browser. Questions like “does the submit button work with good data on all browsers?” are inefficient to answer without automation.
This is a tricky class of test to scope correctly. Automated UI tests are often brittle and hard to change, so if you have too many it can start to feel like the tests are blocking changes. I tend to scope these to the primary sales pipelines and calls to action: “can you succesfully carry out your business?” – anything more than this tends to quickly become more pain than usefulness. Far more efficient to look at how quickly a small mistake in a less important part of your site/application could be fixed.
This is an important problem to take into consideration when deciding where to place business logic. If you have a web application calling a webservice, business logic can be tested behind the service FAR easier than in the web application.
Who writes these?
I’ve usually seen QA’s writing these, although I have written a handful myself in the past. They tend to get written much later in the development lifecycle than other types of test as they rely on attributes of the actual user interface to work. This is the very characteristic which often makes them brittle, as when the UI changes then they tend to break.
How can QA rely on automated UI tests?
Automated UI tests are probably the most brittle and most expensive tests to write, update, and run. It is one of the most expensive ways to find a bug, unless the bug is a regression issue detected by a pre-existing test (and your UI tests are running frequently). To rely on these tests, they need to be used carefully; just test the few critical journeys through your application which can be tested easily. Don’t test business logic this way, ever. These tests are often sat solely in the QA domain; written by a QA, so trusting them shouldn’t be a problem.
This is the one of the few types of testing which human beings are built for. It’s generally applied to user interfaces and is executed by a QA who tries different ways to break the application either by acting ‘stupid’ or malicious. It simply isn’t practical yet to carry out this kind of testing in an automated fashion; it requires the imagination of an actual person. The intention is to catch problems which were not thought of before the application was built. These might be things which were missed, or they may be a result of confusing UX which couldn’t be forseen without the end result in place.
Who does this?
This is (IMO) the ‘traditional’ QA effort.
User Acceptance Testing
Doesn’t UAT stand for “test everything all over again in a different environment”?
I’ve seen the concept of UAT brutalised by more enterprises than I can count. User acceptance testing is meant to be a last, mostly high level, check to make sure that what has been built is still usable when exposed to ‘normal people’ (aka. the end users).
Things that aren’t UAT:
Running an entire UI automation suite all over again in a different environment.
Running pretty much any automation tests (with the possible exception of some core flows).
Blanket re-running of the tests passing in other environments in a further UAT environment.
If you are versioning your applications and tests in a sensible way, you should know what combinations of versions lead to passing tests before you start UAT. UAT should be exactly what it says on the tin: give it to some users. They’re likely to immediately try to do something no-one has thought about – that’s why we do UAT.
Any new work coming out of UAT will likely not be fixed in that release – don’t rely on UAT to find stuff. If you don’t think your pre-UAT test approach gives sufficient confidence then change your approach. If you feel that your integration environment is too volatile to give reliable test results for features, have another environment with more controls on deployments.
Who runs these tests?
It should really be end users, but often it’s just other QA’s and BA’s. I recommend this not being the same people who have been involved right through the build; although some exposure to the requirements will make life easier.
How can QA rely on User Acceptance Testing?
QA should not rely on UAT. By the time software is in a UAT phase, QA should be ok for whatever has been built to hit production. That doesn’t mean outcomes from UAT are ignored, but the errors found during UAT should be more fundamental gaps in functionality which were either not considered or poorly conceived before a developer ever got involved, or (more often than not) yet more disagreement on the colour of the submit button.
Smoke testing originated in the hardware world, where a device would be powered up and if it didn’t ‘smoke’, it had passed the test. Smoke testing in the software world isn’t a huge effort. These are a few, lightweight tests which can confirm that your application deployed properly; slightly more in-depth than a simple healthcheck, but nowhere near as comprehensive as your UI tests.
Who runs these tests?
These should be automated and executed from your deployment platform after each deploy. They give early feedback of likely success or definite failure without having to wait for a full suite of integration tests to run.
How can QA rely on Smoke Testing?
QA don’t rely on smoke tests, these are really more for developers to see fundamental deployment issues early. QA are helped by the early feedback to the developer which doesn’t require them to waste their time trying to test something which won’t even run.
Penetration testing is a specific type of exploratory test which requires some specialist knowledge. The intention is to gain access to server and/or data maliciously via the application under test. There are a few automated tools for this, but they only cover some very basic things. This is again better suited to a human with an imagination.
Who runs these tests?
Generally a 3rd party who specialises in penetration testing is bought in to carry out these tests. That isn’t to say that security shouldn’t be considered until then, but keeping up with new attacks and vulnerabilities is a full time profession.
I haven’t yet seen anyone take learnings from penetration testing and turn them into a standard automation suite which can run automatically against new applications of a similar architecture (e.g. most web applications could be tested for the same set of vulnerabilities), but I believe this would be a sensible thing to do; better to avoid the known issues rather than repeatedly fall foul to them and have to spend time rewriting.
How can QA rely on Penetration Testing?
Your QA team will generally not have the expert knowledge to source penetration testing anywhere other than from a 3rd party. The results often impact Ops as well as Dev, so QA are often not directly involved, as they are primarily focussed on the application, not how it sits in the wider enterprise archetcture.
Bringing It All Together
There are so many different ways to test our software and yet I see a lot of enterprises completely ignoring half of them. Even when there is some knowledge of the different classes of test, it’s deemed too difficult to use more than a very limited number of strategies.
I’ve seen software teams write almost no unit tests or developer written integration tests and then hand over stories to a QA team who write endless UI automation tests. Why is this a bad thing? I think people forget that the test pyramid is about where business value is maximised; it isn’t an over-simplification taught to newbies and university students, it reflects something real.
Here is my list of test types in the pyramid:
Notice that I haven’t included Consumer Contracts in my list. This is because Consumer Contracts can be ran at Unit, Integration, or Component levels, so they are cross-cutting in their way.
In case you need reminding: the higher the test type is on the pyramid, the more expensive it is to fix a bug which is discovered there.
The higher levels of the pyramid are often over-inflated because the QA effort is focused on testing, and not on assuring quality. In an environment where a piece of work is ‘thrown over the fence’ to the QA team, there is little trust (or interest) in any efforts the developer might have already gone to in the name of quality. This leads to inefficiently testing endless combinations of request properties of API’s or endless possibilities of paths a user could navigate through a web application.
If the software team can build an environment of trust and collaboration, it becomes easier for QA to work closer with developers and combine efforts for test coverage. Some business logic in an API being hit by a web application could be tested with 100% certainty at the unit level, leaving integration tests to prove the points of integration, and just a handful of UI tests to make sure the application handles any differing responses correctly.
This is only possible with trust and collaboration between QA and Developers.
Distrust and suspicion leads to QA ignoring the absolute proof of a suite of passing tests which fully define the business logic being written.
What does it mean?
Software development is a team effort. Developers need to know how their code will be tested, QA need to know what testing the developer will do, even Architects need to pay attention to the testability of what they design; and if something isn’t working, people need to talk to each other and fix the problem.
Managers of software teams all too often focus on getting everyone to do their bit as well as possible, overlooking the importance of collaborative skills; missing the most important aspect of software delivery.
Some enterprises have grown their technical infrastructure to the point where dev ops and continuous deployment are second nature. The vast majority of enterprises are still on their journey, or don’t even realise there is a journey for them to take. Businesses aren’t generally built around great software development practices – many businesses are set up without much thought to how technical work even gets done, as this work is seen as only a supporting function, not able to directly increase profitability. This view of technical functions works fine for some time, but eventually stresses begin to form.
Failing at software delivery.
Each area of a young business can be easily supported by one or two primary pieces of software. They’re probably off the shelf solutions which get customised by the teams who use them. They probably aren’t highly integrated; information flows from department to department in spreadsheets. You can think of the integration points between systems as being manual processes.
While the flow of work is funnelled through a manual process such as sales staff on phones or shop staff, this structure is sufficient. The moment the bottleneck of sales staff is removed (in other words, once an online presence is built where customers can be serviced automatically) things need to run a bit quicker. Customers online expect to get instant feedback and delivery estimates. They expect to be able to complete their business in one visit and only expect to receive a follow up communication when there is a problem. Person to person interaction can be very flexible; a sales person can explain why someone has to wait in a way which sounds perfectly fine to a customer. The self-service interaction on a website is less flexible – a customer either gets what they want there and then, or they go somewhere else.
And so businesses start to prioritise new features by how many sales they will bring in, either by reducing the number of customers jumping out of the sales flow or by drawing additional customers into the sales flow.
Problems arise as these features require more and more integration with each of the off the shelf solutions in place throughout the various areas of the business. Tight coupling starts to cause unexpected (often unexplained) outages. Building and deploying new features becomes harder. Testing takes longer – running a full regression becomes so difficult and covers so many different systems that if there isn’t a full day dedicated to it, it won’t happen. The online presence gets new features, slowly, but different instabilities make it difficult for customers to use. The improvements added are off-set by bugs.
Developers find it increasingly difficult to build quality software. The business puts pressure on delivery teams to build new features in less time. The few movements among the more senior developers to try to improve the business’ ability to deliver are short lived because championing purely technical changes to the business is a much more complicated undertaking than getting approval from technical piers. It’s not long before enough attempts to make things better have either been shot down or simply ignored, that developers realise there are other companies who are more willing to listen. The business loses a number of very talented technologists over a few months. They take with them the critical knowledge of how things hang together which was propping up the development team. So the rate of new features to market plummets, as the number of unknown bugs deployed goes through the roof.
It’s usually around this point that the management team starts talking about off-shoring the development effort. The same individuals also become prime targets for sales people peddling their own off the shelf, monolithic, honey trap of a system – promising stability, fast feature development, low prices, and high scalability. The business invests in some such platform without properly understanding why they got into the mess they are in. Not even able to define the issues they need to overcome, never mind able to determine if the platform delivers on them.
By the time the new system is plumbed in and feature parity has been achieved with existing infrastructure, the online presence is a long way behind the competition. Customers are leaving faster than ever, and a decision is made to patch things up for the short term so the business can be sold.
Not an inevitability.
Ok, so yes this is a highly pessimistic, cynical, and gloomy prognosis. But it’s one that I’ve seen more than a few times, and I’m guessing that if you’re still reading then it had a familiar ring for you too. I’m troubled by how easy it is for a business to fall into this downwards spiral. Preventing this is not just about being aware that it’s happening, it’s about knowing what to do about it.
Successfully scaling development effort requires expertise in different areas: architecture, development, operations, whatever it is the business does to make money, people management, leadership, change management, and others I’m sure. It’s a mix of technical skills and soft skills which can be hard to come by. To make things even harder, ‘good’ doesn’t look the same everywhere. Different teams need different stimuli in order to mature, and there are so many different tools and approaches which are largely approved of that one shoe likely won’t fit all. What’s needed is strong, experienced, and inspirational leadership, with buy in from the highest level. That and a development team of individuals who want to improve.
Such leaders will have experience of growing other development teams. They will probably have an online presence where they make their opinions known. They will be early adopters of technologies which they know make sense. They will understand how dev ops works. They will understand about CI and CD, and they will be excited by the idea that things can get better. They’ll want to grow the team that’s there, and be willing to listen to opinion.
Such leaders will make waves. Initially, an amount of effort will be directed away from directly delivering new functionality. Development effort will no longer be solely about code going out the door. They will engage with technology teams at all levels, as comfy discussing the use of BDD as they are talking Enterprise Architecture. Strong opinions will emerge. Different technologies will be trialled. To an outsider (or senior management) it might appear like a revolt – a switch in power where the people who have been failing to deliver are now being allowed to make decisions, and they’re not getting them all right. This appearance is not completely deceptive; ownership and empowerment are huge drivers for team growth. Quality will come with this, as the team learn how to build it into their work from architecture to implementation, and learn how to justify what they’re doing to the business.
Software delivery becomes as much a part of the enterprise as, for example, Human Resources, or Accounts. The CEO may not understand all aspects of HR, but it is understood that the HR department will do what needs to be done. The same level of respect and trust has to be given to software delivery, as it is very unlikely the CEO or any other non-technical senior managers will understand fully the implications of the directions taken. But that’s fine – that’s the way it’s meant to be.
In time to make a difference.
Perhaps the idea of strong technical leadership being critical to technical success is no surprise, it seems sensible enough. So why doesn’t this happen?
There are probably multiple reasons, but I think it’s very common for senior managers to fear strong technical leadership. There seems to be a belief that devolving responsibility for driving change among senior technicians can bring about similar results as a single strong leader while avoiding the re-balancing of power. I see this scenario as jumping out of the same plane with a slightly larger parachute – it’ll be a gentler ride down, but you can only pretend you’re flying. By the time the business makes it’s mind up to try to hire someone, there’s often too much of a car crash happening to make an enticing offering.
If we accept that lifting the manual sales bottleneck and moving to web based sales is the catalyst for the explosion of scale and complexity (which I’m not saying is always the case) then this would be the sweet spot in time to start looking for strong technology leadership. Expect to pay a lot more for someone capable of digging you out of a hole than for someone who has the experience to avoid falling in it to begin with. And other benefits include keeping your customers, naturally.
There are lots of problems that prevent businesses from responding to market trends as quickly as they’d like. Many are not IT related, some are. I’d like to discuss a few problems that I see over and over again, and maybe present some useful solutions. As you read this, please remember that there are always exceptions. But deciding that you have one of these exceptional circumstances is always easier when starting from a sensible basic idea.
Business focused targeting.
For many kinds of work, quicker is better. For software development, quicker is better. But working faster isn’t the same thing as delivering faster.
I remember working as technical lead for a price comparison site in the UK, where once a week each department would read out a list of the things they achieved in the last week and how that had benefited the business. For many parts of the business there was a nice and easy line that could be drawn from what they did each week and a statistic of growth (even if some seemed quite contrived). But the development team was still quite inexperienced, and struggling to do CI never mind CD. For the less experienced devs, being told to “produce things quicker” had the opposite effect. Traditional stick and carrot doesn’t have the same impact on software development as on other functions, because a lot of the time what speeds up delivery seems counter intuitive.
Have two people working on each task (pair programming)
Focus on only one feature at a time
Write as much (or more) test code as functional code
Spend time discussing terminology and agreeing a ubiquitous language
Decouple from other systems
Build automated delivery pipelines
These are just a few examples of things which can be pushed out because someone wants the dev team to work faster. But in reality, having these things present is what enables a dev team to work faster.
Development teams feel a lot of pressure to deliver, because they know how good they can be. They know how quickly software can be written, but it takes mature development practices to deliver quickly and maintain quality. Without the required automation, delivering quick will almost always mean a reduction in quality and more time taken fixing bugs. Then there are the bugs created while fixing other bugs, and so on. Never mind the huge architectural spirals because not enough thought went into things at the start. In the world of software, slow and steady may lose the first round, but it sets the rest of the race up for a sure win.
Tightly coupling systems.
I can’t count how often I’ve heard someone say “We made a tactical decision to tightly couple with <insert some system>, because it will save us money in the long run.”
Please stop thinking this.
Is it impossible for highly coupled systems to be beneficial? No. Is yours one of these cases? Probably not.
There are so many hidden expenses incurred due to tightly coupled designs that it almost never makes any sense. The target system is quite often the one thing everything ends up being coupled with, because it’s probably the least flexible ‘off the shelf’ dinosaur which was sold to the business without any technical review. There are probably not many choices for how to work with it. Well the bottom line is: find a way, or get rid. Ending up with dozens of applications all tightly bound to one central monster app. Changes become a nightmare of breaking everyone else’s code. Deployments take entire weekends. License fees for the dinosaur go through the roof. Vendor lock in turns into shackles and chains. Reality breaks down. Time reverses, and mullets become cool.
Maybe I exaggerated with the mullets.
Once you start down this path, you will gradually lose whatever technical individuals you have who really ‘get’ software delivery. The people who could make a real difference to your business will gradually go somewhere their skills can make a difference. New features will not only cost you more to implement but they’ll come with added risk to other systems.
If you are building two services which have highly related functionality, ie. they’re in the same sub-domain (from a DDD perspective), then you might decide that they should be aware of each other on a conceptual level, and have some logic which spans both services and depends on both being ‘up’, and which get versioned together. This might be acceptable and might not lead to war or famine, but I’m making no promises.
It’s too hard to implement Dev Ops.
No, it isn’t.
Yes, you need at least someone who understands how to do it, but moving to a Dev Ops approach doesn’t mean implementing it across the board right away. That would be an obscene way forwards. Start with the next thing you need to build. Make it deployable, make it testable with integration tests written by the developer. Work out how to transform the configuration for different environments. Get it into production. Look at how you did it, decide what you can do better. Do it better with the next thing. Update the first thing. Learn why people use each different type of technology, and whether it’s relevant for you.
Also, it’s never too early to do Dev Ops. If you are building one ‘thing’ then it will be easier to work with if you are doing Dev Ops. If you have the full stack defined in a CI/CD pipeline and you can get all your changes tested in pre-production environments (even infra changes) then you’re winning from the start. Changes become easy.
If you have a development team who don’t want to do Dev Ops then you have a bigger problem. It’s likely that they aren’t the people who are going to make your business succeed.
Ops do routing, DBA’s do databases.
Your developers should be building the entire stack. They should be building the deployment pipeline for the entire stack. During deployment, the pipeline should configure DNS, update routing tables, configure firewalls, apply WAF rules, deploy EC2 instances, install the built application, run database migration scripts, and run tests end to end to make sure the whole lot is done correctly. Anything other than this is just throwing a problem over the fence to someone else.
The joke of the matter is that the people doing the developer’s ‘dirty work’ think this is how they support the business. When in reality, this is how they allow developers to build software that can never work in a deployed state. This is why software breaks when it gets moved to a different environment.
Ops, DBA’s, and other technology specialists should be responsible for defining the overall patterns which get implemented, and the standards which must be met. The actual work should be done by the developer. If for no other reason than the fact that when the developer needs a SQL script writing, there will never be a DBA available. The same goes for any out-of-team dependencies – they’re never available. This is one of the biggest blockers to progress in software development: waiting for other people to do their bit. It’s another form of tight coupling, building inter-dependent teams. It’s a people anti-pattern.
If you developers need help to get their heads around routing principals or database indexing, then get them in a room with your experts. Don’t get those people to do the dirty work for everyone else, that won’t scale.
BAU handle defects.
A defect found by a customer should go straight back to the team which built the software. If that team is no longer there, then whichever team was ‘given’ responsibility for that piece of software gets to fix the bug.
Development teams will go a long way to give themselves an easy life. That includes adding enough error handling, logging, and resilient design practices to make bug fixing a cinch, but only if they’re the ones who have to deal with the bugs.
Fundamental design flaws won’t get fixed unless they’re blocking the development team.
This isn’t an exhaustive list. Even now there are more and more things springing to mind, but if I tried to shout every one out then I’d have a book, not a blog post. The really unfortunate truth is that 90% of the time I see incredibly intelligent people at the development level being ignored by the business, by architects, even by each other, because even though a person hears someone saying ‘this is a good/bad idea’ being able to see past their own preconceptions to understand that point of view is often incredibly difficult. Technologists all too often lack the soft skills required to make themselves heard and understood. It’s up to those who have made a career from their ‘soft skills’ to recognise that and pay extra attention. A drowning person won’t usually thrash about and make a noise.
A friend tweeted recently about how it isn’t always possible to decide late on which product to use for data storage as different products often force an application to use different patterns. This got me thinking about making other decisions in software design. In general it’s accepted that deciding as late as possible is usually a good thing but I think people often miss-interpret ‘as late as possible’ to mean ‘make it an after-thought’.
‘As late as possible’ is a wonderfully subjective term. It suggests that although we want to wait longer before make a decision, we might not be able to. We might need to make a decision to support the design of the rest of the system. Or perhaps in some cases, making the decision early might be more important than making the ‘perfect’ decision.
I started thinking about how to decide whether a decision should be put off and I was reminded of the work of Roy Osherove. He suggests that development teams transition through different states and should be managed differently in each state. There is a similar methodology which relates the approach to software design with different categories of problem space. It’s called Cynefin (pronounced like ‘Kevin’ but with a ‘n’ straight after the ‘K’).
To quote Wikipedia:
The framework provides a typology of contexts that guides what sort of explanations or solutions might apply.
There’s a diagram that goes with this and helps give some context (thanks Wikipedia):
I don’t want to do a deep dive into Cynefin in this article (maybe another day) but to summarise:
Obvious – these solutions are easy to see, easy to build, and probably available off the peg. They’re very low complexity and shouldn’t require a lot of attention from subject matter experts.
Complicated – these solutions are easier to get wrong. Maybe no-one on the team has implemented anything similar before, but experience and direction is available possibly from another team.
Complex – these solutions have lots of possible answers but it’s unclear which will be best. While this is new to your company, other companies have managed to build something similar so you know it is possible.
Chaotic – these solutions are totally new and you have no point of reference as to what would be the best way to implement. You may not even know if it’s possible.
In general, an enterprises core domain will sit in both Complicated and Complex. Chaotic might be something you do for a while but the focus is to move the solution back into one of the other categories.
So what does this have to do with making decisions?
Well, I suggest that the decision making process might change depending on what category your solution falls into.
Obvious – obvious solutions probably don’t have many difficult decisions to make. This is not your core domain (unless you work in a really boring sector) so throwing money and resources at obvious solutions is not sensible. The driver for choosing tech stack might well be just “what’s already available” and might be a constraint right up front. You may want to buy an off the shelf product, in which case a lot of decisions are already made for you. If SQL Server is the quickest path to delivery then it might well be the right thing to use here, even if you’re longing to try a NoSQL approach.
Complicated – complicated solutions are often solved by taking advice from an expert. “We found a separate read concern solved half our problems.” and “A relational database just doesn’t work for this.” are both great nuggets of advice someone who’s done this before might put forward. These solutions are in your core domain, you want to avoid code rot and inflexible architectures – deciding late seems generally sensible, but the advice from your experts can help scope those decisions. Focus on finding the abstractions on which to base the solution. You might know that you’ll need the elasticity that only the cloud can provide, but you might leave the decision on which provider until as late as possible.
Complex – complex solutions are where experts are harder to get involved. They might be in different teams or hired from a consultancy. The focus should still be on finding the right abstractions to allow critical decisions to be delayed. Running multiple possible solutions in parallel to see what works best is a great approach which will give your team confidence in the chosen option. A subject matter expert might be more useful in explaining how they approached defining a solution rather than just what the solution was.
Chaotic – it might seem a terrible idea to try and make decisions in this situation but there are advantages. Chaotic can become Complex if you can find an anchor for the solution. “How do we solve this with ‘x’?” is a lot easier for a team to decide than a more general approach. You’ll almost certainly want to run with two or three possible options in parallel. Keep in mind that whatever decision you make may well eventually be proved incorrect.
I think this shows how the approach to decision making can be affected by what category of solution you’re working on. By picking the right strategy for the right kind of problem, you can focus resources more cost effectively.
A particularly tricky epic hits the development team’s radar. The Product Manager has been mulling it over for a while, has a few use cases from end users and has scoped things pretty well. He’s had a session with the Product Owner who has since fleshed things out into an initial set of high priority stories from a functional point of view. The Product Owner spends an hour with the Technical Lead and another Developer going over the epic. It quickly becomes apparent that they’re going to have to build some new infrastructure and a new deployment pipeline so they grab one of the Architects to make sure their plans are in line with the technical roadmap. Some technical stories are generated and some new possibilities crop up. The Product Owner takes these to the Product Manager. Between them they agree what is expected functionally and re-prioritise now they have a clearer picture of what’s being built. In the next grooming session the wider team hit the epic hard. There’s a lot of discussion and the Architect gets called back in when there are some disagreements. The QA has already spent some time considering how testing will work and she’s added acceptance criteria to all the stories which are reviewed by the devs during the meeting. Each story is scored and only when everyone is happy that there is a test approach, a deployment plan and no unresolved dependencies, does a story get queued for planning. It takes three sprints to get the epic completed, during which the team focus primarily on that epic. They pair program and work closely with the QA and Product Owner who often seem like walking, talking specifications. There are a few surprises along the way and some work takes longer than expected but everyone’s ok with that. Some work gets completed quicker than expected. The Product Manager has been promoting the new feature so when it finally goes fully live the team get some immediate feedback on what users like and dislike. This triggers a few new stories to make some changes that go out in the next release.
Well, that all sounded really simple. Everyone involved did their bit, no-one expected anything unreasonable, the only real pressure came from the team’s own pride in their work and everyone went home on an evening happy. So why is it that so often this isn’t the case? Why do some teams or even entire companies expend huge amounts of effort but only succeed in increasing everyone’s stress levels, depriving people of a decent night’s sleep and releasing product with more bugs than MI5?
If you’re reading this then hopefully team management is something you’re interested in and you’re probably well aware that there is never just one thing that makes it work. In fact, Roy Osherove described three distinctly different states that a team can find itself in which require very different management styles and expose very different team dynamics.
Often team members are affected by influences external to the team – for example, the Product Manager is almost certain to be dealing with multiple teams and end users. Their day could have started pretty badly and that will always colour interactions with others – we’re only human. So at any one given moment of interaction, the team could be in one of many states that could either be conducive to a good outcome or that could be leading to a problem.
Let’s try pulling apart this idealistic scenario and see how many opportunities there were for derailment.
In our scenario, the epic isn’t straight forward so the Product Manager has been thinking about what it means to his end users and where it fits into his overall plan. A lesser Product Manager might not have given it any attention yet and instead of having some stories outlined there might not be much more than a one line description of the functionality. Lack of care when defining work speaks to the rest of the team about how little the epic matters. If the Product Manager doesn’t care enough about the work to put effort into defining it properly, then others will often care just as little.
Early Architectural Input
Right at the beginning the Architect is questioned about how they see the work fitting in with the rest of the enterprise. Without talking tech early the team could waste time pursuing an approach which is at best not preferred and possibly just won’t work.
Product Owner and Technical Lead
The Product Owner and the Technical Lead take on the initial task of trying to get to grips with the story together. These two are a perfect balance of product and development. Moving up the seniority tree, the Product Manager and the Architect can often disagree vehemently, but the less senior pair need the relationship of mutual trust they’ve built. Nowhere is there a better meeting of the two disciplines. Lose either of these roles and the team will suffer.
After looking at things from a technical point of view, the Product Owner goes back to the Product Manager to discuss some changes. If the Product Owner isn’t open to this then many opportunities for quick wins will be missed and it becomes much more likely that the team will be opening at least one can of worms that they’d prefer not to.
Although strictly speaking all the work that goes into defining a story is ‘grooming’ and doesn’t have to include the whole team, there should be a point (probably in a grooming session) where the majority of the team gets the chance to review things and give their own opinions. If this doesn’t happen then much of the team’s expertise is being wasted. Also, some team members will be specialists in certain areas and may see things that haven’t yet been considered. Lastly (and maybe most importantly) if the team are simply spoon fed a spec and expected to build it, they aren’t being engaged in the work – they are far more likely to care less about what is being built.
Ready for Planning
The team make sure that the stories have test approaches, reams of acceptance criteria, no unresolved dependencies and that everyone believes the stories are clear enough to work on them. This is a prerequisite to allowing the work to be planned into a sprint. Without this gate to progression, work can be started that just can’t be finished. Working on something that can’t yet be delivered is a waste of time.
Questions with Answers
During the sprints, the Product Owner and QA are on hand all the time for direct face to face discussions about the work. “Does this look right?” “Can we reduce the number of columns on the screen?” “Is this loading quick enough?” – developers are at the sharp end and need to have answers right there and then so they can build. Without Product and QA representation available, a simple question like these can stall a story for an hour, a day or maybe too long to complete in the sprint.
This is one small, quite contrived scenario. In real life things are rarely as straightforward but with every interaction and every piece of work within the team and beyond there is the risk for some kind of derailment. To list every scenario would take forever, so how can problems be avoided?
Knowing Their Jobs
Each individual in this team knew where they fitted into the puzzle. Everyone worked together as a team. In fact things need to go a little further than that for the best outcome – team members should know what each other’s jobs entail. Take the Technical Lead role; they should be protecting their team from ill conceived ideas (even when there seems to have been a lot of effort gone into those ideas). It’s part of their job to question whether proceeding with a given piece of work is the best thing to do. When they do raise concerns, these should be listened to and discussed with a level of maturity and mutual respect that befits someone who fulfilling one of the requirements of their job. Equally, when a QA suggests that a feature seems flawed even though it passes acceptance criteria, their issue should be addressed, even if nothing is ultimately changed. This is part of the QA’s job – raising issues of quality.
I’d like to outline what I see as an excellent balance of team members and responsibilities. I don’t mean to insinuate that this is the only way a team can work – in fact I encourage every team to find their own way. Regular retrospectives where the team look at how they’re performing, what works and what doesn’t, allow the team to form their own processes. This is nearly always preferable to mandating how things should be done as people are more likely to stick to their own ideas.
That having been said, I believe the lines are there to blur and if I were to define roles around those lines, it would be like this:
The Product Manager is responsible for the product road map. They are focussed on what the end users are missing in the product as it stands and they are working to define feature sets that will better meet the end user’s requirements and better any competition. It’s a trade off between effort, risk, how badly the feature is wanted, when it is technically feasible to implement it.
It is vitally important that their roadmap is fully understood initially by the Product Owner and then by the senior members of the team. Decisions are constantly being made during delivery which benefit from a knowledge of product focus.
The Product Manager is not responsible for writing stories or working with the development team beyond receiving demo’s of completed or near completed work. They’re not responsible for the velocity of the team or for managing the team in any way, although they do have a vested interest in when things are going to get delivered and will almost certainly require an explanation if things can’t or don’t happen as quickly as they’d like.
The Product Owner can best be thought of as an agile Business Analyst. They are responsible for communicating the vision of the Product Manager to the team. They sit within the team and are answerable to the Team Lead rather than the Product Manager. This might seem odd but the team needs them to put their need to understand the product ahead of anything else. Without good understanding of what is being built, the team cannot hope to build the right thing.
The Product Owner wants their stories to be understood, so they will want the team to be involved in deciding what information is recorded in a story and how it is formatted. Story design will usually evolve over a few sprints from discussions during retrospectives. It is the Product Owner’s sole responsibility to make sure that all functional details are described once and only once and that none are left out. They will often work closely with the team’s QA to make sure all functional aspects have acceptance criteria before the story is presented in a grooming session. At a minimum, before a story is allowed to be planned into a sprint, all functional details must have an acceptance criteria to go with them. It is quite acceptable for these acceptance criteria to be considered the specification, rather than there being two different descriptions of the same thing. This reduces duplication, which is desirable because duplication leaves a story open to misunderstanding.
The QA should be focussing on automation as much as possible. The last thing a QA wants to do is spend their time testing – it isn’t a good use of their time. Yes, there are a few things that are still best done by opening the app and having a look and I’m not saying that automation should be solely relied on, but it should be the default approach. Recognising that something can’t be automated should be the exception rather than noticing that something could. Something automated never needs to be tested again until that thing changes. Something manual adds to the existing manual set of tests. Only so much of the latter can go on before the QA has no time to really think about the quality of the product.
The QA should be happy that they understand how they are going to ensure the quality of every single story picked up by the team. They should have recorded acceptance criteria which act as a gateway to ‘done’ for the developers and a reminder for the QA about different aspects of the story.
Sometimes there will be several different levels of criticality to functionality. For example, imagine a piece of work on a back office system which includes modifications to an integration to a customer facing system. It’s not the end of the world if the back office system needs a further tweak to make things absolutely perfect, but if the integration breaks, then end users may be affected and data may be lost. The integration should definitely be tested with automation – for every field and every defined piece of logic involved. The internal system’s ui, could probably be tested manually, depending on the circumstances.
The QA should make sure they spend time with the Product Owner looking at new stories and working quite abstractly to define acceptance criteria around the functionality. During grooming, technical considerations will kick up other acceptance criteria and a decision around what needs to be automated and what doesn’t can be made by the team. A decision which the QA should make sure is made actively and not just left to a default recipe.
The Architect is not really a part of a specific team. They will have their own relationship with the Product Manager and will be familiar with the product road map. They will have three main considerations on their mind:
They will be looking at the direction that technology in general is going and will be fighting the rot that long lived systems suffer from. This isn’t because code breaks, quite the opposite – the problem is that working code could stay in place for years after anyone who had anything to do with writing it has left the company, even if that code is not particularly good. Replacing working systems is a difficult thing for businesses to see a need for but if the technology stack that is relied on isn’t continuously refreshed then it becomes ‘technical debt’. Something that no-one wants to touch and that is expensive to fix.
They will be making sure the the development teams have enough information about the desired approach to system building. If following a DDD approach, the teams need to know what the chosen strategy is for getting data from one subdomain to another. They want to know what format they should be raising their domain events in, how they should be versioned. Given new ideas and technologies, they need to know when it’s ok to start using them and more importantly, when it’s too premature to start using them.
In conjunction with the Product Roadmap, they will be defining the Technical Roadmap. This document takes into consideration what Product are wanting to do and what the technical teams are wanting to do. It’s almost a given that the Technical Roadmap will have to feedback into the Product Roadmap as it shows what needs to be done to deliver. For this reason, it’s generally a good idea not to consider a Product Roadmap complete until it has been adjusted to accommodate the technical plan.
In the scenario at the start of this post, the Architect was consulted because additional infrastructure was going to be needed. This is something that could happen several times a week and something that the Architect should be prepared for. They need to give definite answers for what should be done right now, not what that decision would be if the question was asked in a month’s time – this leads to premature adoption of new concepts and technologies that aren’t always fully understood or fully agreed on.
The Developers are at the sharp end. They should have two main focusses, and make no mistake, both are as important as each other:
They need to build the Product.
They need to work out how they can become better developers.
Notice how I haven’t mentioned anything about requirements gathering. This is something that has classically been considered what most developers overlook, but in a development team this is done by the Product Owner, Product Manager and to a lesser extent the QA. It’s this support net of walking talking specifications combined with well defined stories that allow a developer to focus on doing what they do best, writing code.
Something that can be ignored by less experienced developers is that the Product is not just the code that makes it up; it has to be delivered. If building a web application then this will ultimately have to deployed, most likely into at least one test environment and into production. Maybe this process is automated or is handed on to a ‘Dev Ops’ team, but it’s still down to the developers to build something that can actually be deployed. For example, if a piece of code that would normally talk to a database is changed so that it now only works with a new version of the database, how do we deploy? If we deploy the database first, then the old code won’t work when it calls it. If we deploy the code first then the call to the old database won’t work. There could be dozens of instances of the code, which could take hours to deploy to one by one – it isn’t usually possible to deploy data changes at exactly the same time as deploying code. Basically, the Developers have to keep bringing their heads out of the code to think about how their changes impact other systems.
Technical Lead is a role that is very close to my heart as it’s a role I tend to find myself in. I believe that if a development team delivers a product, then a Technical Lead delivers the team. They are technically excellent but are split focussed – they are as much about growing the team as they are about making sure the team are delivering what is expected.
There is a lot of noise about a good Technical Lead being a ‘servant leader’ and for the most I think this is true. Still, there are times when it’s necessary to be direct, make a decision and tell people to do it. When to use each tactic is something only experience teaches but get it wrong and the team will lose focus and cohesion.
It’s quite common to find that a Technical Lead has one or more developers in their team that are better coders than themselves. It’s important for someone in the lead role to realise that this is not a slur on their abilities; the better coder is a resource to be used and a person to grow. They aren’t competition. This often happens when developers don’t want to split their focus away from the technology and onto the people. They become incredible developers and eventually often move into a more architectural role. Because of their seniority, they can also stand in for the Technical Lead when it’s needed. Which can allow a lead to occasionally focus on building something specific (it’s what we all enjoy doing, after all).
Ultimately it’s down to the Technical Lead to recognise what is missing or lacking and make sure that it is made available. They see what the team needs and makes sure the team gets it (even if sometimes that’s a kick up the arse).
Blurring the Lines
Not every team has enough people to fill all these roles. Not every company is mature enough to have the staff for each of these roles. So this is where the lines begin to blur.
It isn’t always critical to have each of these roles fulfilled as a full time resource. Each company and team’s specific situation needs to be considered on it’s own merits. What is critical is understanding why these roles exist. This knowledge allows the same individual to have different hats at different times. But beware of giving too much power to one person or to removing the lines of discussion which is what generates great ideas.
So What Actually Makes it Work?
In a highly technical environment I find my own opinion on this very surprising. I don’t think it’s any individual’s technical ability or how many different programming languages they can bring to bear. I don’t think it’s having every detail of every epic prepared completely before development work starts. Primarily, consideration must be given to the structure and dynamics of a team. the above is an excellent starting point although as I’ve already mentioned, it may be that multiple roles are taken by any one individual. Other than this, the I believe a successful team will have these qualities:
Patience, mutual respect and a healthy dose of pragmatism.
If there is one constant in every team I’ve ever worked with or in, it’s that some things will go wrong. Mistakes will happen – no-one is immune (and if they claim otherwise don’t believe them). Making progress forwards doesn’t mean always building the most incredibly perfect solution, it means getting product out the door which end users will be happy with. So if there are mistakes, have the patience to realise that it was inevitable. If you make the mistake, have the respect for the rest of your team to realise that they can help fix things – believe that they aren’t judging you by your mistake. Finally, remember that the perfect solution is not the perfect system – there’s more to delivering software than trying to find all the bugs.
A team with these values will ultimately always outperform a collection of technically excellent individuals.