Code News

Silicon Valley's Immortalists Will Help Us All Stay Healthy

Testing Data-Intensive Code With Go, Part 5

  • Overview

    This is part five out of five in a tutorial series on testing data-intensive code. In part four, I covered remote data stores, using shared test databases, using production data snapshots, and generating your own test data. In this tutorial, I'll go over fuzz testing, testing your cache, testing data integrity, testing idempotency, and missing data.

    Fuzz Testing

    The idea of fuzz testing is to overwhelm the system with lots of random input. Instead of trying to think of input that will cover all cases, which can be difficult and/or very labor intensive, you let chance do it for you. It is conceptually similar to random data generation, but the intention here is to generate random or semi-random inputs rather than persistent data.

    When Is Fuzz Testing Useful?

    Fuzz testing is useful in particular for finding security and performance problems when unexpected inputs cause crashes or memory leaks. But it can also help ensure that all invalid inputs are detected early and are rejected properly by the system.

    Consider, for example, input that comes in the form of deeply nested JSON documents (very common in web APIs). Trying to generate manually a comprehensive list of test cases is both error-prone and a lot of work. But fuzz testing is the perfect technique.

    Using Fuzz Testing 

    There are several libraries you can use for fuzz testing. My favorite is gofuzz from Google. Here is a simple example that automatically generates 200 unique objects of a struct with several fields, including a nested struct.  

    import ( "fmt" "github.com/google/gofuzz" ) func SimpleFuzzing() { type SomeType struct { A string B string C int D struct { E float64 } } f := fuzz.New() uniqueObject := SomeType{} uniqueObjects := map[SomeType]int{} for i := 0; i < 200; i++ { f.Fuzz(&object) uniqueObjects[object]++ } fmt.Printf("Got %v unique objects.\n", len(uniqueObjects)) // Output: // Got 200 unique objects. } Testing Your Cache

    Pretty much every complex system that deals with a lot of data has a cache, or more likely several levels of hierarchical caches. As the saying goes, there are only two difficult things in computer science: naming things, cache invalidation, and off by one errors.

    Jokes aside, managing your caching strategy and implementation can complicate your data access but have a tremendous impact on your data access cost and performance. Testing your cache can't be done from the outside because your interface hides where the data comes from, and the cache mechanism is an implementation detail.

    Let's see how to test the cache behavior of the Songify hybrid data layer.

    Cache Hits and Misses

    Caches live and die by their hit/miss performance. The basic functionality of a cache is that if requested data is available in the cache (a hit) then it will be fetched from the cache and not from the primary data store. In the original design of the HybridDataLayer, the cache access was done through private methods.

    Go visibility rules make it impossible to call them directly or replace them from another package. To enable cache testing, I'll change those methods to public functions. This is fine because the actual application code operates through the DataLayer interface, which doesn't expose those methods.

    The test code, however, will be able to replace these public functions as needed. First, let's add a method to get access to the Redis client, so we can manipulate the cache:

    func (m *HybridDataLayer) GetRedis() *redis.Client { return m.redis }

    Next I'll change the getSongByUser_DB() methods to a public function variable. Now, in the test, I can replace the GetSongsByUser_DB() variable with a function that keeps track of how many times it was called and then forwards it to the original function. That allows us to verify if a call to GetSongsByUser() fetched the songs from the cache or from the DB. 

    Let's break it down piece by piece. First, we get the data layer (that also clears the DB and redis), create a user, and add a song. The AddSong() method also populates redis. 

    func TestGetSongsByUser_Cache(t *testing.T) { now := time.Now() u := User{Name: "Gigi", Email: "gg@gg.com", RegisteredAt: now, LastLogin: now} dl, err := getDataLayer() if err != nil { t.Error("Failed to create hybrid data layer") } err = dl.CreateUser(u) if err != nil { t.Error("Failed to create user") } lm, err := NewSongManager(u, dl) if err != nil { t.Error("NewSongManager() returned 'nil'") } err = lm.AddSong(testSong, nil) if err != nil { t.Error("AddSong() failed") }

    This is the cool part. I keep the original function and define a new instrumented function that increments the local callCount variable (it's all in a closure) and calls the original function. Then, I assign the instrumented function to the variable GetSongsByUser_DB. From now on, every call by the hybrid data layer to GetSongsByUser_DB() will go to the instrumented function.     

    callCount := 0 originalFunc := GetSongsByUser_DB instrumentedFunc := func(m *HybridDataLayer, email string, songs *[]Song) (err error) { callCount += 1 return originalFunc(m, email, songs) } GetSongsByUser_DB = instrumentedFunc

    At this point, we're ready to actually test the cache operation. First, the test calls the GetSongsByUser() of the SongManager that forwards it to the hybrid data layer. The cache is supposed to be populated for this user we just added. So the expected result is that our instrumented function will not be called, and the callCount will remain at zero.

    _, err = lm.GetSongsByUser(u) if err != nil { t.Error("GetSongsByUser() failed") } // Verify the DB wasn't accessed because cache should be // populated by AddSong() if callCount > 0 { t.Error(`GetSongsByUser_DB() called when it shouldn't have`) }

    The last test case is to ensure that if the user's data is not in the cache, it will be fetched properly from the DB. The test accomplishes it by flushing Redis (clearing all its data) and making another call to GetSongsByUser(). This time, the instrumented function will be called, and the test verifies that the callCount is equal to 1. Finally, the original GetSongsByUser_DB() function is restored.

    // Clear the cache dl.GetRedis().FlushDB() // Get the songs again, now it's should go to the DB // because the cache is empty _, err = lm.GetSongsByUser(u) if err != nil { t.Error("GetSongsByUser() failed") } // Verify the DB was accessed because the cache is empty if callCount != 1 { t.Error(`GetSongsByUser_DB() wasn't called once as it should have`) } GetSongsByUser_DB = originalFunc }Cache Invalidation

    Our cache is very basic and doesn't do any invalidation. This works pretty well as long as all songs are added through the AddSong() method that takes care of updating Redis. If we add more operations like removing songs or deleting users then these operations should take care of updating Redis accordingly.

    This very simple cache will work even if we have a distributed system where multiple independent machines can run our Songify service—as long as all the instances work with the same DB and Redis instances.

    However, if the DB and cache can get out of sync due to maintenance operations or other tools and applications changing our data then we need to come up with an invalidation and refresh policy for the cache. It can be tested using the same techniques—replace target functions or directly access the DB and Redis in your test to verify the state.

    LRU Caches

    Usually, you can't just let the cache grow infinitely. A common scheme to keep the most useful data in the cache is LRU caches (least recently used). The oldest data gets bumped from the cache when it reaches capacity.

    Testing it involves setting the capacity to a relatively small number during the test, exceeding the capacity, and ensuring that the oldest data is not in the cache anymore and accessing it requires DB access. 

    Testing Your Data Integrity

    Your system is only as good as your data integrity. If you have corrupted data or missing data then you're in bad shape. In real-world systems, it's difficult to maintain perfect data integrity. Schema and formats change, data is ingested through channels that might not check for all the constraints, bugs let in bad data, admins attempt manual fixes, backups and restores might be unreliable.

    Given this harsh reality, you should test your system's data integrity. Testing data integrity is different than regular automated tests after each code change. The reason is that data can go bad even if the code didn't change. You definitely want to run data integrity checks after code changes that might alter data storage or representation, but also run them periodically.

    Testing Constraints

    Constraints are the foundation of your data modeling. If you use a relational DB then you can define some constraints at the SQL level and let the DB enforce them. Nullness, length of text fields, uniqueness and 1-N relationships can be defined easily. But SQL can't check all the constraints.

    For example, in Desongcious, there is a N-N relationship between users and songs. Each song must be associated with at least one user. There is no good way to enforce this in SQL (well, you can have a foreign key from song to user and have the song point to one of the users associated with it). Another constraint may be that each user may have at most 500 songs. Again, there is no way to represent it in SQL. If you use NoSQL data stores then usually there is even less support for declaring and validating constraints at the data store level.

    That leaves you with a couple of options:

    • Ensure that access to data goes only through vetted interfaces and tools that enforce all the constraints.
    • Periodically scan your data, hunt constraint violations, and fix them.    
    Testing Idempotency

    Idempotency means that performing the same operation multiple times in a row will have the same effect as performing it once. 

    For example, setting the variable x to 5 is idempotent. You can set x to 5 one time or a million times. It will still be 5. However, incrementing X by 1 is not idempotent. Every consecutive increment changes its value. Idempotency is a very desirable property in distributed systems with temporary network partitions and recovery protocols that retry sending a message multiple times if there is no immediate response.

    If you design idempotency into your data access code, you should test it. This is typically very easy. For each idempotent operation you extend to perform the operation twice or more in a row and verify there are no errors and the state remains the same.   

    Note that idempotent design may sometimes hide errors. Consider deleting a record from a DB. It is an idempotent operation. After you delete a record, the record doesn't exist in the system anymore, and trying to delete it again will not bring it back. That means that trying to delete a non-existent record is a valid operation. But it might mask the fact that the wrong record key was passed by the caller. If you return an error message then it's not idempotent.    

    Testing Data Migrations

    Data migrations can be very risky operations. Sometimes you run a script over all your data or critical parts of your data and perform some serious surgery. You should be ready with plan B in case something goes wrong (e.g. go back to the original data and figure out what went wrong).

    In many cases, data migration can be a slow and costly operation that may require two systems side by side for the duration of the migration. I participated in several data migrations that took several days or even weeks. When facing a massive data migration, it's worth it to invest the time and test the migration itself on a small (but representative) subset of your data and then verify that the newly migrated data is valid and the system can work with it. 

    Testing Missing Data

    Missing data is an interesting problem. Sometimes missing data will violate your data integrity (e.g. a song whose user is missing), and sometimes it's just missing (e.g. someone removes a user and all their songs).

    If the missing data causes a data integrity problem then you'll detect it in your data integrity tests. However, if some data is just missing then there is no easy way to detect it. If the data never made it into persistent storage then maybe there is a trace in the logs or other temporary stores.

    Depending how much of a risk missing data is, you may write some tests that deliberately remove some data from your system and verify the system behaves as expected.

    Conclusion

    Testing data-intensive code requires deliberate planning and an understanding of your quality requirements. You can test at several levels of abstraction, and your choices will affect how thorough and comprehensive your tests are, how many aspects of your actual data layer you test, how fast your tests run, and how easy it is to modify your tests when the data layer changes.

    There is no single correct answer. You need to find your sweet spot along the spectrum from super comprehensive, slow and labor-intensive tests to fast, lightweight tests.  

    1 day 9 hours ago

How Bossip Smashed Headline Conventions to Smithereens

The Whirl Is Not Enough: Hundred-Dollar Fidget Spinners

Manipulating HTML5 Canvas Using Konva: Part 5, Events

  • If you have been following this series from the beginning, you should now be very comfortable with shapes, groups, and layers. You should also be able to easily draw some basic and complex shapes on the canvas using Konva. If you plan on using Konva to create some interactive app or games, learning how to bind events to different shapes on the stage is the next logical step.

    In this tutorial, you will learn how to bind events to any shape using Konva. You will also learn about event delegation and propagation. Sometimes, you need might need to control the hit region of a shape as well as fire events programmatically. We will be discussing these two topics as well.

    Binding Events to a Shape

    You can bind different events to any shape created using Konva with the help of the on() method. All you have to do is pass the name of the event as the first parameter and the function to be executed when the event occurs as the second parameter. You can use Konva to detect mouseup, mousedown, mouseenter, mouseleave, mouseover, mousemove, click, and dblclick. Additionally, Konva allows you to detect wheel, dragstart, dragmove, and dragend events.

    Here is an example that detects mousedown and mouseleave events on a regular polygon (hexagon). Similarly, the smaller circle is bound to the mouseover and mouseup events and the larger circle is bound to the mouseenter, mouseleave, and mousemove events.

    var canvasWidth = 600; var canvasHeight = 400; var stage = new Konva.Stage({ container: "example", width: canvasWidth, height: canvasHeight }); var layerA = new Konva.Layer(); var polyA = new Konva.RegularPolygon({ x: 125, y: 125, sides: 6, radius: 80, fill: "yellow", stroke: "black", strokeWidth: 5 }); var circA = new Konva.Circle({ x: 275, y: 225, height: 100, fill: "orange", stroke: "black" }); var circB = new Konva.Circle({ x: 475, y: 275, radius: 100, fill: "red", stroke: "black" }); layerA.add(polyA, circA, circB); stage.add(layerA); polyA.on("mousedown", function() { polyA.sides(polyA.sides() + 1); layerA.draw(); }); polyA.on("mouseleave", function() { var totalSides = polyA.sides(); if(totalSides > 3) { polyA.sides(polyA.sides() - 1); } layerA.draw(); }); circA.on("mouseover", function() { circA.strokeWidth(10); layerA.draw(); }); circA.on("mouseup", function() { circA.strokeWidth(5); layerA.draw(); }); circB.on("mouseenter", function() { stage.container().style.cursor = "crosshair"; }); circB.on("mouseleave", function() { stage.container().style.cursor = "default"; }); circB.on("mousemove", function() { var pointerPos = stage.getPointerPosition(); var r = pointerPos.x % 255; var g = pointerPos.y % 255; circB.fill("rgb(" + r + ", " + g + ", 100)"); layerA.draw(); });

    If a user presses any mouse button while the cursor is inside the regular polygon, we increase the number of sides of the polygon by 1. The sides() method can be used without a parameter to get the number of sides for a polygon or used with one parameter to set the number of sides for a polygon. You can also get the number of sides using getSides() and set the number of sides using setSides(). The sides of the polygon are reduced by one whenever the mouse cursor leaves the polygon.

    For the smaller circle, the mouseover event is used to set the stroke width value to 10. The mouseup event changes the stroke width value to 5. Keep in mind that the mouseup event has to occur inside the circle itself. For example, the stroke width will not change to 5 if you press the mouse button inside the circle and then release it only after the cursor is outside the circle.

    In the case of the larger circle, we are using the mousemove event to change its fill color. We are also changing the cursor of the larger circle using stage.container().style.cursor whenever the cursor moves in and out of the circle.

    One important thing that you should keep in mind is that you have to call the draw() method on the respective layer if the event listeners for any shape resulted in a change of attributes like fill color, stroke width, etc. Otherwise, the changes won't be reflected on the canvas.

    You don't have to bind one event at a time to a shape. You can also pass in a space delimited string containing multiple event types to the on() method. This will bind all the events listed in the string to that particular shape.

    Konva also supports corresponding mobile versions of all these events. For example, you can register touchstart, touchmove, touchend, tap, dbltap, dragstart, dragmove, and dragend using Konva on mobile devices.

    You can also fire any of these events for a particular shape or shapes using the fire() method. Similarly, Konva allows you to fire custom events like throwStones.

    Removing Event Listeners

    You can remove any event listeners attached to a shape with the help of the off() method in Konva. You just have to specify the event name that you don't want to listen to.

    You can also create multiple event bindings for a single shape. For example, let's say you have a circle and you want to increase the radius of the circle every time the mouse cursor goes over it, up to a certain limit. You might also want to change the fill color of the circle on every mouseover event. 

    One option is to do both these tasks inside a single mouseover event listener and stop updating the radius later. Another option is to create two mouseover event listeners with different namespaces to identify them. This way, you will be able to increase the radius and change the fill color independently.

    circA.on("mouseover.radius", function() { var curRadius = circA.radius(); if(curRadius < 150) { circA.radius(curRadius + 5); layerA.draw(); } else { circA.off('mouseover.radius'); } }); circA.on("mouseover.fillcolor", function() { var h = Math.floor(Math.random()*360); var color = "hsl(" + h + ", 60%, 60%)"; circA.fill(color); layerA.draw(); });

    You should note that I have added layerA.draw() inside both listeners. If you fail to add it inside the mouseover.fillcolor listener, the color will stop updating as soon as the radius becomes 150.

    Instead of removing one event listener at a time, you can also stop listening to all events bound to a shape using the setListening() method. You can pass true and false to this method in order to turn event listeners on and off. Keep in mind that you will also have to redraw the hit graph of the affected layer by calling the drawHit() method right after you call setListening().

    Event Delegation and Propagation

    Instead of binding events directly to all the shapes present on a layer, you can also bind an event to the layer itself. After that, you can determine which shape triggered the event using the target property of the event object. This way, Konva allows you to effectively delegate events from the parent to its children.

    Let's say you are listening to click events on a circle drawn on a layer in Konva. The same click event propagates to the containing group as well as the containing layer. This may or may not be the intended behavior. If you want to prevent the event from bubbling up inside a shape to the containing layer, you can set the cancelBubble property for the event object to true.

    var canvasWidth = 600; var canvasHeight = 400; var stage = new Konva.Stage({ container: "example", width: canvasWidth, height: canvasHeight }); var layerA = new Konva.Layer(); var circA = new Konva.Circle({ x: 300, y: 200, height: 100, fill: "orange", stroke: "black", name: "Orange Circle" }); var starA = new Konva.Star({ x: 125, y: 125, innerRadius: 25, outerRadius: 75, rotation: 90, fill: "blue", stroke: "black", name: "Blue Star" }); var ringA = new Konva.Ring({ x: 475, y: 275, innerRadius: 25, outerRadius: 75, fill: "brown", stroke: "black", name: "Brown Ring" }); var textA = new Konva.Text({ text: "", fontFamily: "Calibri", fontSize: 24, fill: "black", x: 10, y: 10 }); layerA.add(circA, starA, ringA, textA); stage.add(layerA); layerA.on("click", function(e) { var shapeName = e.target.attrs.name; textA.setText(shapeName); layerA.draw(); });

    I have used the name property to assign a name to each of our shapes. The setText() method is then used to change the text inside textA to the name of the shape we just clicked.

    Custom Hit Regions

    In the above example, the ring registered a click on it when the click occurred between the inner and outer circle. What if you wanted to register the click inside the smaller circle as well? Konva allows you to define custom hit regions using the hitFunc property. This property accepts a function as its value, and this function is used to draw the custom hit region.

    The following example shows you how to create custom hit regions. You should now be able to click in the area between the star spikes and still register a click. With the help of custom hit regions, you can make sure that your users don't have to click on exact locations to register a click event. This can result in a better user experience when dealing with smaller or more complex shapes.

    var starA = new Konva.Star({ x: 125, y: 125, innerRadius: 25, outerRadius: 75, rotation: 90, fill: "blue", stroke: "black", name: "Blue Star", hitFunc: function(context) { context.beginPath(); context.arc(0, 0, this.getOuterRadius(), 0, Math.PI * 2, true); context.closePath(); context.fillStrokeShape(this); } }); var ringA = new Konva.Ring({ x: 475, y: 275, innerRadius: 25, outerRadius: 75, fill: "brown", stroke: "black", name: "Brown Ring", hitFunc: function(context) { context.beginPath(); context.arc(0, 0, this.getOuterRadius(), 0, Math.PI * 2, true); context.closePath(); context.fillStrokeShape(this); } });

    Final Thoughts

    In this tutorial, we have covered different mobile and desktop events that you can bind to any shape in Konva. You can attach these events one at a time or many of them at once. Konva also allows you to fire your own custom events programmatically using the fire() method. The last section of the tutorial showed you how to define your own hit regions in order to detect hits on an area that could be larger or smaller than the original shape.

    Combining the knowledge of this tutorial with others in the series, you should now be able to draw a variety of shapes on the canvas and allow your users to interact with them. 

    If you have any questions related to this tutorial, feel free to let me know in the comments.

    1 day 11 hours ago

How Ford Build a New Kind of Engine for Its GT Supercar

10 Best Android App Templates for Business

  • Business apps for Android are a constantly growing market thanks to a community of avid developers and the increasing popularity of mobile devices.

    It is no surprise, therefore, that business app templates are also in demand as they help cut down some of the tedious parts of coding and allow developers to focus on the more interesting work of making their app unique.

    Today I’ll be looking at 10 of the best business app templates for Android developers to be found at CodeCanyon.

    The app templates I’ve chosen relate to all aspects of business, from increasing productivity to keeping on top of expenses and increasing a company’s visibility in an ever more crowded marketplace.

    1. Productivity Timer

    Time is a precious commodity and no more so than in business, where there are always a million people and pressing tasks vying for attention. It stands to reason that if you want to get things done, you need to carve out specific blocks of time when you can focus single-mindedly on the task at hand. This is where the Productivity Timer comes in. 

    This template is designed for developers who want to create a productivity app to help users carve out blocks of time to concentrate on important tasks. Users set a timer for a specific task, and during that period they can choose to disable features on their phone such as sounds, vibration, and Wi-Fi so they can avoid interruptions and give the task at hand their full attention.

    2. Expense Manager

    Keeping track of our expenses helps us have better control over our money, but it can be a real pain. Expense Manager is an app template for designers who want to create an efficient financial tracker. It is designed to organize income and expenses, and record movement of money by date. 

    Users can review reports on their finances over a day, a week, a month, or a year. The app can remind users to pay bills and can also help them to manage receipts so that getting reimbursements for business expenses is easy.

    3. Biz Tool

    With so much international exchange, most business people need to move seamlessly between different cultures and systems. This is where an app like Biz Tool is indispensable. 

    The Biz Tool template allows developers to create an app that combines a number of essential functions in one handy app. It has a calculator function and can convert units like mass, length, area, speed, temperature, volume, and even currency.

    4. Business Card Maker

    Business Card Maker is one great idea in a beautifully designed app template. Developers can use the template to create an app that allows users to make their own digital business cards which they can share with colleagues or clients digitally, thus eliminating the need and expense of creating a physical card. 

    The app contains a number of card templates that are easy for users to customise according to their taste and requirements.

    5. Inventory Management App

    The Inventory Management App template helps developers create an app that will track a business’s stock inventory, check which products are running low, and send out notifications of low inventory levels.

    It is a very useful tool for businesses to stay on top of their stock inventory, reduce paperwork, and manage their orders and suppliers more efficiently.

    6. Events App Template

    One thing a good business has to do is communicate, and with the Events App Template developers can create an app that allows business users to store and share information about upcoming events with clients and colleagues alike. 

    Users just need to press a button in order to add events they’re interested in to their native Calendar app or to open an event’s address in Google Maps to get directions. They can also share news about the event on Facebook, Twitter, mail, messenger or other social apps they use.

    7. Conference App

    Developers can use the Conference App template as a skeleton or ready-made mobile app for businesses that want to share information about internally organised conferences and meetings within and outside of their organisation.  

    The app will act as a mobile guide for guests, giving them a variety of information about the conference including the goals of the conference, the day’s agenda, the speakers, sponsors, location, etc.

    8. My Business App

    My Business App helps developers create apps that users can use to turn their website into an app. And why would a business want to do this, you might ask? Well, mobile apps are actually a simplified version of websites and thus are more purpose-focused and easier to use. 

    They also offer a more interactive experience and more brand awareness because users see the company’s logo on their screen each time they unlock their smartphone. In addition, companies can use push notifications to inform users of updates and special features or send custom messages. What’s great about this is that push notifications have an open rate of 90%, which is higher than any other current communication method. 

    9. Business Portfolio App

    The Business Portfolio App is a great way for businesses to market themselves to prospective clients, by presenting a portfolio of their business in a modern and innovative way. 

    The app template can be customised to present whatever information the business wants to highlight, including team information, a portfolio of successful projects, clients worked for, testimonials, contact details, and social media addresses.

    10. News App Template

    Keeping on top of the news—particularly as it pertains to business—is critical to every business, and so a great news app is a crucial asset. News App Template allows developers to create their own News App with headlines, breaking news labels, various categories, deep link sharing, video and image support, and much more. 

    What’s great is that the app can be customised to pull news from pre-determined sources so that users only get news that is relevant to their needs.

    Conclusion

    These 10 best business app templates for Android are just a small selection of hundreds of Android app templates we have available at CodeCanyon, so if none of them quite fits your needs, there are plenty of other great options to choose from.

    And if you want to improve your skills building Android apps and templates, then check out some of the ever-so-useful Android tutorials we have on offer.

    1 day 12 hours ago

Bitcoin and Blockchain Will Reveal What They're Actually Good For in 2018

The Hard Math Behind Bitcoin's Global Warming Problem

Pages