Shifting left your UI tests | Arjan Blok | #SeConfLondon

Shifting left your UI tests | Arjan Blok | #SeConfLondon

Articles Blog


OK everybody, does any of you knoe this guy? This is can you name him, Carlos? Bob Ross. It’s Bob Ross. He used to have a television show on PBS in
which within half an hour, he would paint a beautiful picture. Now, I listened to his show. I watch his show and I think the idea was
that you could paint along with him. I just listened to his soothing voice and the wisdoms
that he shared. Now, I used some of those wisdoms in my talk. That’s why I would like to rename the title
to the joy of component testing. Because his show was called the joy of painting. Now, he recently had his Internet revival. So, any of his episodes you can watch on Twitch
and on YouTube. For the few people who have no idea who I’m
talking about. I’ll let the man introduce himself.>>I’m Bob Ross. Certainly, glad you could join me today. Fantastic day here. And a fantastic painting. If this is your first time with us, I send
a personal invitation that you bring out the paints and follow along. I will show you how to create some of the
most fantastic paintings you have seen in just a few minutes. Let’s go to the canvas and get started. ARJAN: Yes, let’s get started. I worked at a company called Mendix. We build the beam is pretty bright. We build a load code application platform
to allow developers to visually create, deploy and run their apps. We recently well, four years ago we started
building Webb IDE, which is what you’re currently looking at. This is a single page application built with
TypeScript, which is a super set of JavaScript, to add typings. And it’s using React as a frontend. Now, this application, of course, had to be
tested. We decided to, well, use a Selenium based
solution. WebDriverIO. It’s a custom client side implementation of
the Selenium protocols. We did this because, well, what gives better
confidence than seeing your application being automated just as if a user would, you know. We also we were pretty new as a team with
React. So, it was our first experience. We didn’t know how to do things better. Or different. And while I had quite some experience with
Selenium, using it in previous projects. So, you know, we went with what we knew. Also, the developers were busy setting up
the project and I had time to set up the test automation. Yes. Now, one of Bob Ross’ quotes is if you do
too much, it’s going to lose its effectiveness. So, think of a painting, right? Bob Ross could paint beautiful trees. And whenever he painted a single tree could
be a majestic thing on a painting. But as soon as you start adding millions and
millions of trees in our case you get just a blank or a green square. Which is not very interesting. Not very useful either. You wouldn’t hang that on your wall. Now, the same holds for our test automation. Basically, we started writing UI tests. We wrote a lot of them. And everyone every one of them had a purpose. But all combined together, the down sides
started out weighing the benefits. So, basically, we wrote too many UI tests
as we also saw yesterday, the experts agree on. Because this is what Simon Tweeted. I love what we can do with UI tests. But each one you write needs to be incredibly
valuable to offset the cost of writing and maintaining it. We noticed it. It was getting slower and slower, it took
more time to run every time. Now, of course, this is all relative. But we had as a goal to make sure we could
continuously deliver our product. So, we were deploying to production every
time. And our UI tests, a few hundred of them, took
about a half an hour to run. And that was the cutoff point. We already paralyzed that with using Selenium. But it was growing. And that was annoying. We had a few flakey tests. And, again, this was not like a lot of tests
per build. But there were the occasional flakey tests
which, you know, causes us to lose a bit of confidence in the product. This is not necessarily Selenium’s fault. Because it does what you tell it to. But writing at that scale, writing writing
proper automation is difficult. And also, when something failed, when a test
failed, it was pretty difficult to analyze the failures. Because since we’re testing from the user
perspective, we are basically testing our entire stack. Now, I think this is Bob Ross’ most famous
quote. We don’t make mistakes. We just have happy accidents. He used this whenever, you know, he was painting
a picture, and something went wrong. He he dropped some paint. He would turn that paint into a beautiful
tree or one of his famous ones. Now, we painted something for ourselves. I know, I’m not a fan of the pyramid either
or the ice cream cone or whatever. But it serves a purpose here. We painted this for ourselves. Basically, well, I don’t know if I don’t
think I need to explain it too much. But, you know, the higher up you go, the slower
your test execution will be. The higher up you go, the higher the test
integration. Testing things as a bigger scope, things will
be less prone to flakiness. When something fails, it’s more difficult
to determine the root cause of that. We needed to basically fix our mistake and
turn it into a happy accident. The way we did this was by shifting left. So, if you take that ice cream cone, turn
it on its side and put our pipeline. I know it’s an oversimplified reality. But basically, this was our situation. We had a bunch of UI tests. They were taking a very long time to run. There were also, well, more integrated. So, they’re less likely to fill. What we did was basically we started moving
UI tests to the other parts. So, as a very simple example, we had a UI
test that tested all the possible values for an integer in the UI. Now, this was the the rule for that was
actually just in a JavaScript method. So, we decides to replace that UI test or
like nine out of the ten iterations of it with a unit test. And we also started to do component testing. Now, component testing, or component tests,
you heard it a few times already before, these are basically unit tests for your UI. So, we added that layer. It’s just above the unit tests. But before running the actual application. And we fixed our mistake and made it into
our happy accident. So, the reason we could add component tests,
as I mentioned, was because we were using React as a frontend framework. This also holds for Angular and Vue. Basically, there are a few benefits to using
a component based UI framework like this. You write a component once and you can reuse
it multiple times in your application or even within multiple applications. These components are consistent. You’re reusing the same ones. They’re also highly collaborative. It means you’re creating small visual components. So, they’re easy to reason about what they
should do, how they should react, what they should look like with your product owner UX. But also, with your testers. So, they’re highly testable. This is a set which is not necessarily the
best setup or the only way to do this. There are a limited amount of variations here. But we decided to use gist as a test runner. A framework like Mocha or TestNG. We have Enzyme from Airbnb which gives us
the opportunity to load components and interact with them as if we were doing UI tests. And we used JS DOM to render those components
in memory, basically. As we saw with the pyramids, the component
testing is just above the unit testing level. Maybe even the same, depends on who you ask
and what pyramid you look at. But the idea is that since it’s lower in the
pyramid, it’s faster. And in our case, we didn’t need a real browser. We had a minimal stripped down browser in
the form of JS DOM. We don’t need a running application, which
is a big bonus because, well, that means you can earlier on in your pipeline run your test. But also, you know, just like with other unit
tests, you can run them in watch mode while you’re developing on the items. On the components. And there was more isolation. You know, since we’re only testing a small
part, there’s less that can go wrong. And once it goes wrong, it’s easier to analyze
where it actually went wrong. Maybe some of you are familiar with the set,
maybesome of you aren’t. I’ll try to create some context. Is this readable? Okay. Yeah. This is a button component. It takes the properties in state. Initially the state will contain the property
color, which is green. And the render function returns the actual
component. In this case, a button. This button shows the text that we pass in
when weinstantiate it this component. There’s two methods that has a background
color which is determined bit color in the state. And the on click method will execute the function
below and that will change from the color from green to red and vice versa. So, when you actually mount in component and
run it in a real browser, this is what it looks like. You know? You have a button with the text “Hello world.” That’s what we’re supplying. And when you click it, it changes color. So, how would you go about and test this? Well, I made a small test just to show you
what we can do. It’s basically so, we use Enzyme. We mount the actual component. We instantiate the component. Supply it with some text that we want. And then the first test basically looks at
if the text was actually found in the real component. And the second test actually checks the state
color. And then interacts with the button. So, it actually clicks the button. And verifies that the state has been changed. So, not too different from your Selenium tests. Yes. Bob Ross always used to well, of course,
he painted. I’m goingtry something similar. No videos or other. Just do it live. So, this is the application that we built
that you’re looking at. So, let’s say I would want to go to a domain. I want to direct my an entity on it. Doesn’t really matter. The context is not that important. But the idea is that if I fill in a character
that’s not allowed here and I press “Create,” an error message should show up. So, we add these tests, right? In our UI tests. This kind of test, this level. Well, we can actually do this in our component
tests. As we’ll try to show you. Yes. Let me first run it because the first time
always takes a little bit longer. So, I’ll show you the test is this. Is this readable or is this too small? Okay. So, well, there is some step setup which we’ll
not dive into too much. But basically, as just before with the button,
here we are creating or instantiating that create entity dialogue. We can access the dialogue, so we don’t have
to navigate to it first. Then we can interact with this dialogue. So, here we store the results in a wrapper. And we interact with the wrapper by finding
the input and assimilating a change on it. We’re changing the text value of that input. And then we we’re clicking the primary
button, which is the create button in this case. And then we expect that there will be an element
with the entity or with the with the with the class T error message and we expect
that to match with this error message. So, if I go back to my test, this just ran. First time always takes a little bit longer. Just is flakey that way. Or weird that way. It holds stuff in memory. I don’t know what it does. But the actual the point I’m going say
is that that I wanted to show you is that this test running this test takes for
mounting the component, interacting with it and actually unmounting it takes under 62
milliseconds. Now, if we run that same test as our with
our UI test, we will, one, need a will the more setup. We are starting the browser, downloading some
stuff. We are creating a project for ourselves for
this application. And then so, the browser will start. It will have to navigate to the actual environment. To the actual document that we want to test. And need to create an entity. And then the pop up box will appear. And then we will interact with the pop up
and validate that the error message is there. Now, of course, this is not also not the
fastest implementation. I don’t know. It tries to access the Internet as well. Which might be a little bit slow. And it’s doing a lot of setup. And it’s not like, you’re not going to
run a single test like this when you’re actually testing, you know? You’re going to have multiple tests in a file. But to show you the difference, the actual
iteration of this test took almost 20 seconds. And we’re basically testing the same thing. Did the element appear in the DOM? So, there’s a great speed improvement there
already. Another great thing that becomes available
when you do component testing is snapshots. Yeah. A snapshot is basically a you generate
a snapshot of the DOM once and you can then compare it many times. So, more generic. You can use it as a more generic assertion. Instead of having specific checks on specific
element on specific error messages, you can also test the complete part of the DOM. Maybe the entire DOM of your component. But it actually works for all serializable
JavaScript objects. So, I have a demo of the snapshot. If we go back to our our component test. So, instead of having expect to equal here. What we could also do is to match snapshot. And now, first time we run this
it will take it will take a snapshot and I’ll show you the content of it. Will hopefully run a bit faster now as well. Or not. Right. So, now took 200 milliseconds which is still
yeah. Still quite fast. Let me see. I need to go here. Shortcut, shortcuts. So, it created a snapshot which is what we’re
currently looking at. The test title is here. And the expected message is here. So, now every time you run it, it will elevate
this expected messages there. But what we can also do let’s close this. This. So, instead of to match the snapshot, instead
of matching the specific element, which contains the T error message attribute clause, what
if we match the entire wrapper? So, if we run this test again, it should fail
because we’re no longer finding that specific error message. We’re actually looking for the entire DOM
now. Of the component. So, this simple pop up is actually pretty
massive. You’ll see sort of a get comparison where
this is what it was looking for. The first part. And the second one is what it actually received. Now, it’s pretty easy to say, hey, this is
actually what we want. This is correct. Run it again, with in mind the view flag. And now it will accept this as the correct
answer. Answer. At the correct expected result. Right. There are a few best practices when using
snapshots. You should always treat your snapshots as
code. Don’t just, you know, glance at them and say,
ah, that’s fine, right? Also, watch out for update fatigue, you know,
if you’re constantly working on your component and you think, yeah, that’s fine. Update, update, update. You might miss something. Your test should be deterministic. So, if you have a timer in there or a random
value, every time you run it, your test will basically fail. So, there are ways to strip that out. And you should always use descriptive test
names, right? The intent of the test should always be clear. Because since we removed that error message
from your test, the test has become less clear, I would say. You know? You’re not sure what you’re exactly looking
for. Now, it’s in the snapshot file. But try to make clear from your tests what
you should actually be what the intent of the actual test is. So, this all sounds pretty great, right? We have lots and lots of we can have lots
and lots of fast tests. We can have a pretty broad checking mechanism. Why would I still be using Selenium? Now, I think you’ve heard this message already
a few times before during this conference. People are slowly moving away from it. For good reasons. As aye shown you. But it doesn’t mean it doesn’t hold its place. You should just use it wisely. It’s a real browser, right? It’s what your customers are actually looking
at. Nobody uses JS DOM to navigate websites. I don’t think that’s even possible. It supports multiple browsers. Chrome, Firefox, we all know. And it’s actually its purpose is actually
testing our integration, you know? We can test our whole application as a whole
which means behavior of the entire application might differ from, well, your components only
describe certain behavior. And your entire application describes your
application behavior. There’s the application state. So, if you change a component or fill in some
text or click a button on the right, something on the left might accidently start changing. You’re not gonna check and get that if you’re
doing component testing only. And there’s the CSS component. The CSS components can interact with each
other. Causing undesirable, well, errors, basically. So, yeah, that was it. There’s basically that’s what we’ve been doing. And just recently, one of our new QA’ers,
she started experimenting with visual differences. Trying to find those in our test suite. So, I would like to show you that. And it’s not it’s not so far it’s not
so different from doing snapshot testing, right? It’s broadening your check, basically, to
encapsulate more than just a specific message. So, if we go back to our UI test. And let’s create let’s duplicate this test. We did this because we were having a few issues
with things that didn’t work correctly, but Selenium couldn’t find, you know? Once a button is behind another button or
behind another element, then Selenium will fail. I cannot click it, right? But sometimes something else happens. I’ll try to show you. So, instead of having this specific check,
let’s do… we’re going to check the dialogue. The root element is something we made specifically
in our framework. It’s basically getting the actual web element
instead of the instance of the clause that describes the object. We need to give it a tag. So, let’s call it create dialogue name and
the file will be saved as. And then we expect it to… to equal… all
right. In this case, we expect it to equal zero. We expect it to be pixel perfect. Now, there are better ways to do this, but
just for the example I’m going to show you, pixel perfectness. So, once again, if we run this UI test, this
Selenium test, and, of course, normally you wouldn’t do that check probably twice. You know? Now we’re testing with the we’re basically
doing the same test twice. But there will be a difference. So, the first time you run it, just as with
the snapshot test, it will create a component. Or it will create a snapshot. In this case, a screenshot. I’ll try to show you when it’s done. So, first time it navigates, the incorrect
text. And the second time, it does the same. But instead of validating a message, it takes
the screenshot. So, if I go to my browser, it basically
this is the screenshot it created. And that’s what we expect to be there again
next run. Now, as an example I will change my branch. Right. So, what I did, I’ll show you in the actual
application. You can see, there’s the create button here. Now, I changed it. I’ll reload my application. When I now create one, the button is gone. Not really. The button is actually still there. It’s just 1×1 pixel big. So, well, I guess you can all figure out what
will happen now. But I’m still going to run it anyway. 12 minutes left. So… So, again, first test. Opens browser, navigates to our desired state. Then fills in the text, fixed the but then
and validates the error message. Second test, does the same. Creates the test data, navigates to the desired
state. Fills in the text. Creates a screenshot. And now, hopefully, yes, we have one passing
test and one failing test. And hopefully if I picked the right ones,
yes. This is the image that we expected there to
be. This is the one that was found. After the button is clicked it becomes active. And this is, of course, the difference that
has been shown. Now, we use the library made by Vim who is
unfortunately not here. But you can use Applitools for this. There are multiple ways to do this. So, before I reach the end of my talk, I want
to share one one more thing about Bob Ross. Each episode he created three paintings. One he did beforehand and put to the side
as a reference during the show. So, the second one he was creating live, or
live. It was recorded, of course. But on the show, he was creating the second
one. And there was a third one that he created
afterwards, and they took pictures from him. From him while he was creating this. So, you know, even Bob Ross didn’t do it correctly
the or straight good the first time. He improved on himself. He iterated over his paintings. And I hope this talk inspires you to do the
same. Look a bit further than your default methods. Because that’s what we did. You know, we started off with writing lots
and lots of UI tests until it basically started failing on us. We looked for better ways, better alternatives. And it’s this second iteration our last one? Probably not. Maybe I’ll be here next year talking about
some AI stuff that we implemented. I heard some awesome talks about that. So, who knows? But I encourage you, you know, we don’t really
know where this goes. I’m not sure we really care. If something pop up, we’ll adapt. You know? We’ll change. We’ll make our process better. And I hope you do too. Because if you do, then you, too, can paint
almighty pictures. Thank you.

Leave a Reply

Your email address will not be published. Required fields are marked *