D3.js + Websocket for live Web applications
Not all JS is bad. True story!
Loïc Hoguin - @lhoguin
Erlang Cowboy and Nine Nines Founder
Background
LeoFS sponsored project
- LeoFS developers and QA team need tools to inspect their systems
- Users need tools to help debug issues with LeoFS
- A centralized tool would help greatly
- Alien Shaman is a project to make it happen
Inspired by Bigwig
- Bigwig, Spawnfest 2011 winner, was a good start
- Unfortunately was never maintained (grrrr!)
- Also was local-only, not distributed
Not a fork of Bigwig
- Modifying Bigwig would be more work than not
- Distribution means events and UI need to be decoupled
- Event/probe library is Alien
- Web UI is Shaman
Alien
- Event/probe library
- Different probes: ondemand, inline and process
- Routing and filtering rules
- Allows grouping events into batches (minimal impact on the running system)
- Can send events through Erlang, TCP, UDP, you name it
- Most of the work is done
Shaman
- Web UI
- Forward events to UI using Websocket
- Can handle many types of events, even custom ones
- Pluggable interface
- Live interface, no database required
- Work in progress!
Events storage
- Events should be stored on disk for later retrieval
- Alien relays can be used for this
- Shaman will need an interface to preload data/load older data
- This isn't even a WIP yet, but it will come eventually
Alien Shaman demo
D3.js
Data-Driven Documents
- Javascript library for manipulating documents
- Bind data to the DOM
- Apply transformations to it based on data
- All using standard HTML, SVG and CSS
Standards-driven
- You are not limited to what the library offers
- Use all the features offered by browsers as soon as they come out
- Easy to debug using browser inspectors
What can you do with D3.js?
- Update existing data on a page
- Generate HTML tables
- Draw diagrams using SVG
- Create complex animated representations of your data
To this end, D3.js provides
- Selectors
- Dynamic properties
- Enter and exit selections
- Transitions
Enter and exit
- Nodes are bound to data
- Data passed to D3 with no corresponding node: enter
- Data passed to D3 with corresponding node: update
- Other nodes with no corresponding data: exit
__data__
- Data bound to nodes is stored in the __data__ property of the node
- It can be retrieved, manipulated, updated
- (It is generally not needed to do so)
- Selectors return a list of nodes so it makes it easy to access too!
Protocol
Events forwarding
- Websocket frame contains list of events
- Events encoded using msgpack
- Websocket connection simply encodes and forwards events
Client commands
- Client may send commands
- Example: nodes.connect to connect to a node
- Will be used to dynamically connect to nodes and install probes on them remotely
- Commands are always ran from the Websocket process and may return errors to the UI
- Client doesn't really need to wait for completion, an event will arrive instead
Identifying frames
- All frames, regardless of origin, have one common key
t
- This is the type of the frame
- Events are of type
data
- Each
data frame also has a name in key n and actual data in key d
Dynamic JS event handlers
- The JS code registers handlers for each
t
- But also for each type of data
n
- Custom handlers can be registered easily
Receiving events from JS
- Receive Websocket frame
- For each event, find the handler for
t and pass it the event
- If the event is
data, find the handler for n and pass it the data d
- Let D3.js magically process this data and update the page
Specially crafted HTML...
Plus a simple mapping function make...
A developer really bored
- No specific code needed to update pages
- Rows are created automatically
- Cells get the expected data values automatically
- Boring means good!
This is truly powerful
I mean, seriously, powerful
- Adding new information is a two step process
- Modify Erlang code to send more data
- Write a little HTML where values will be displayed
- No Javascript!
- *applause*
And this is just the beginning
- What about client-side calculations?
- Data that is shown in different places?
- And graphs? Show me pretty colors!
Client-side calculations
Make the client do the work
- Avoid doing calculation in the Erlang node
- Only do if it means sending a lot less data
- Let the client perform the calculation
- Example: difference of a value between T and T+1
Tricks required
- D3.js helps very little here
- It's still possible by accessing the data directly in the HTML elements
- Accessing the data like this is documented, so it's still good
Update __data__ in the DOM
Multiple destinations
Use case: modals
- I have a table with general information about processes
- I want to be able to click a row and display a modal with extra data
- Problem: data is tied to the rows
Modal solution
- Creating/updating HTML for every possible modal is madness
- We need to retrieve data directly from the row elements and assign that to the modal
- That operation needs to be done each update
Graphs
Accumulate data
- Graphs typically show data across a period of time
- We need to accumulate data
- Everytime we get an event we push that data to the graph
- And then update the graph
Library of graphs
- Graphs can have a very similar interface regardless of how they display data
- We can have a library of reusable graphs
- We can even give the option to change the type of graph displayed in one click
- Still a work in progress
Future of Alien Shaman
To start
- Replicate Bigwig/Observer functionality
- We need to see processes, applications, etc.
- We want some general view of all the nodes (example: memory usage)
- We want a common interface for receiving and displaying logs
Tracing
- We want to be able to trace systems
- We want to be able to trace not just one node, but follow messages across nodes
- Thankfully Erlang comes with distributed tracing tools!
- (Phew!)
Attach to any node
- We want to be able to use probes even on systems without Alien loaded
- That means we'll need to send the whole application using distribution and start it
- It also means that for this operation we need to have the Erlang distribution running, it's not required once configured though
And the most important part
- Get a nice logo!
- Make it pretty!
- Use special effects!
I think we're done
