A group of 5 Clojure enthusiasts from Solita visited the Clojure eXchange conference in London on December 3rd and 4th. Clojure eXchange, or ClojureX, was organized for the 8th time now and this year’s event was the largest in its history. The conference was single-track and spanned two full days of talks on all things Clojure.
The conference was organized by Skills Matter at Codenode. Signal was kind enough to sponsor the conference to avoid lack of conference t-shirt. There weren’t any other sponsors, but the conference itself was organized in a very professional manner.
The talks were very pragmatic, with many first-hand experiences on solving real world problems with Clojure. Despite the pragmatic focus, it was neat to see the continued tradition of having some of the more abstract tar pits explored as well ;)
Day 1
The welcoming ceremony had everyone stand up, and gradually sit down based on how long they had been using Clojure. This revealed us the following facts:
- More than half of the audience had over 3 years of Clojure experience
- Christophe Grand didn’t understand the instructions, or claimed the seniority by being the last-man-standing at the 10+ year mark*
This continues to make the case for Clojure’s maturity and production viability.
[*] We are inclined to believe in the latter
Keynote: The Spirit of Clojure
Karsten Schmidt kicked off the conference with a keynote filled with visual programmatic art.
Karsten has no fear of diving into new languages while solving problems in unfamiliar areas, reading research papers, writing programs in literate style and learning new tooling along the way (Emacs!). This is quite a feat to do :) Clojure made him feel like a novice again, a feeling that resonates well with many in the community.
Karsten, the author of the Thi-ng family of libraries, showcased many illustrative projects created with it, using formulas like De Jong attractors, Hilbert curve, Gyroid formula and many others. Volumetric visualizations allow sampling the formulas to generate views in different resolution, which allows you to not think in polygons. Karsten had also done data visualization projects and it was neat to see what the visual presentation of the data tells even without labeling.
He ended with a couple of notes: One on how people in different language communities share same ideas but use different names (Clojurians seldom mention Sparql but talk about Datomic, GraphQL is known in Javascript land but not Datomic). Another point he made was on the difficulty of writing performant code in Clojurescript, when immutability is the default, since one has to write more code to “undo” immutability.
In the latter point, it might be better to write such tight loops in plain Javascript. A runtime that does this for you would probably be a worthy research subject.
The keynote was interesting but there was quite a big difference between what the title made it sound like it was and what the talk itself ended up being.
Polymorphism à la Clojure: Functional Object Oriented Programming is not an Oxymoron.
Yehonathan Sharvit, author of klipse (executable slides via klipse), presented the Oxymoron (e.g. “The painting is an original copy”) of OOP in functional style.
He did an exploration into Object Oriented Programming style first via plain maps and then using protocols. In the first example, he used a send-msg function to talk to the objects which are a map with data and operations and returning a new object on accepting the message since mutation is bad: “When a person drinks, he’s not the same person” :). In this style of intermediate results, threading macro is your friend.
The protocol example used defrecord for implementing a protocol, which get’s rid of a send-msg. He presented five commandmends as a view to OOP in Clojure.
Couple of takeaway points:
- Beware that
dissocmakes a record into a map. - Although records cannot be used as functions, one can implement the
IFnon a record viaextend-type.
Data Science in Languages I Don’t Understand
Rob Stanley gave a speech on an implementation of a data driven system for helping farmers all around the world to perform their jobs more efficiently.
A big part of the software was Natural Language Processing, namely, detecting the language and intent of questions sent by farmers using SMS. Implementation consisted of Clojure code for the mathematical models, and a custom EDN formatted DSL for defining patterns that are used for the actual intent prediction.
A general feeling from the speech; Data Scientists are capable of learning and using Clojure (who would have known?), but Data Science in Clojure still requires writing a lot of your own plumping.
Now you’re Speaking my Language! : Building, Maintaining and Using a Patient-Friendly Medical Ontology
Chloe Pont and Sam Oleary from Health Unlocked showed how they used Clojure in building a social health discussion website.
The domain deals with text processing (synonyms, misspellings, acronyms and abbreviations) and a medical hierarchy of treaments and conditions. The main part of the talk was the ontology used to power the website.
The ontology is stored in a Clojure map (presumably in an EDN file), updated via code scripts and the resulting map is diffed via editscript library, which yields changes that are applied to a relational database in which the result is stored in a table structure, served via an API (a Clojure app) for use by the site. They later made a dsl to limit and verify changes to apply in the ontology change scripts. The ontology is used on the site for suggesting tags for posts, autocompletion and widening search by term detection.
The last question from the audience was most revealing: What database was used? The answer, MySQL, triggered spontaneous laughs from the audience :). Apparently the company previously had a Neo4J based system that was bloated and instead of extending that fragile system, they decided to reimplement what they actually needed in a simpler way, which seems fair enough.
Lightning Talk: Building A Simulation-Based Text Adventure
Peter Westmacott showcased a text-based browser game he’s been writing with ClojureScript. Being highly systems-driven akin to titles like Dwarf Fortress and Prison Architect, the game runs a simulation of a village and its inhabitants with the aim of modelling social interactions and exploring the concepts of contagious violence and scapegoating. Peter found Clojure’s declarative nature, as in open maps and data type literals, a boon for this kind of exploratory simulation work. Other topics such as obscure French philosophical anthropology were also covered, as per usual in a FP conference. Peter also managed to cram a flashy demo in his 10-minute talk, which was much appreciated by the crowd.
Lightning Talk: Running without an API
In this lightning talk Gaivile Vezeviciute presented an application that showed whether a running event in a running community called ParkRun was cancelled. The premise was that the events were announced on the community website, but there was no way to be notified about cancellations other than checking the website listing all the events. To solve this problem, Gaivile developed a Clojure application to check the status of the events she was interested in. As there was no API to check this information, her application scraped the website and checked the status of the events given as a parameter. The app got its start at London Clojure Dojo, an event where people gather to level up their Clojure skills through practical exercises.
This talk was a good fit for the lightning talk format. The problem statement was easy to explain and understand and there was enough time to present the solution in proper detail.
Lightning Talk: Making Music With Alda & Edna
Another eloquent speaker, Tristram Oaten demoed Alda and Edna, music programming languages for musicians and Clojurians. Tristram started with a brief intro on music theory and moved on to building progressively more complex compositions using an edn syntax. What stood out the most was the clever usage of Clojure data structures to express musical concepts such as chords.
Learning How to Design Automatically Updating AI With Clojure, Kafka and Deeplearning4j
Jason Bell is an author and at least locally known speaker since, starting at the apology at the start of the talk, half of the audience would miss the jokes :).
Jason went through an example of piping events through kafka and training models with Deeplearning4j (neural network), Weka (decision tree) and Apache Commons Math (linear regression). The example project was quite event-based, with separate topics for predictions requests and responses, and a Kafka Streams application for branching messages by content (training data/build model command) from an event topic.
Some libraries spotted were a Deeplearning4j wrapper jutsu.ai, Weka wrapper clj-ml (although it is a bit dated) and kixi.stats.
Are You Writing Java in Clojure?
Erik Assum’s talk was one (of many) that we most waited for.
With a background on working on legacy code bases and seeing code which is hard to test or reason about, Erik offered stable advise on how to structure your code to maximize reasonability and testability.
Erik had many good references in his talk, one of which was Stratified Design from the SICP book, describing how such design doesn’t necessarily mean a 3-tier architecture. He also referred to the imperative shell/functional core idea from Gary Bernhardt’s Boundaries talk. Make sure to reserve time to go over the references, they are good quality and well summarized in the talk.
With the help of the Elements of Clojure book he descibed solid ways to structure code to separate pull/transform/push phases. He also showed bad code that his boss wrote, how often would you do that in a conference? :)
Speed Bumps Ahead
Alexander Yakushev, who among other things, maintains the Clojure goes fast! website, gave a good presentation on performance optimization.
He had an interesting example where he compared the way Clojure compiler uses reflection to plain direct reflection use. It turned out that in this example, reflection itself wasn’t the main offender, but the Clojure compiler implementation, which is forced to loop over all methods of a class to find one that matches the argument types at runtime.
Even more interestingly, he used clj-java-decompiler, a library which decompiles Clojure code into semi-readable Java, to narrow down to the Clojure compiler implementation that revealed the above fact.
Alexander is the author of many useful libraries found in the clojure-goes-fast Github organization, such as clj-memory-meter and clj-async-profiler. Another interesting take was writing code that requires more tight control over performance in Java and using virgil to still keep the REPL workflow (one doesn’t have to quit the repl to recompile/load the Java classes).
Lightning Talk: Beginners Introduction to Clara Rules
Charlotte Fereday from ThoughtWorks (our Apax sibling) presented clara-rules, an expert system/rule engine library for managing complex and volatile business logic. Clara takes in a set of forward chaining rules or conditions in form of if A then B and then evaluates facts against them. The purpose of a rule engine is to simplify writing business logic into code. While probably overkill for simple cases, the library might work wonders in untangling convoluted conditionals into simple self-contained rules. Unlike enterprise rule systems, Clara offers a lightweight developer-driven approach based on functional programming, with all rules and facts stated out in plain Clojure.
Lightning Talk: Monitoring Clojure Applications with Prometheus
Joachim Draeger from the conference’s sponsor Signal, demonstrated monitoring Clojure microservices via Prometheus, an open source monitoring toolkit originated at Soundcloud. Joachim started by stressing the importance of visibility and metrics in distributed systems and then went on to shed light on the approach they’re taking at Signal. The presented solution seemed relatively plug-and-play: pull in a Ring middleware from the Clojure Prometheus client library, configure it to expose a metrics API endpoint and point your Prometheus instance to it. Prometheus then takes care of scraping data from the endpoint and feeding it onto a neat Grafana dashboard. The client library supports all sorts of custom metrics but also includes pre-made solutions for exposing JVM and Ring handler metrics, which seemed very compelling.
Brewing CIDER: It Starts with an Orchard
Bozhidar, the community’s beloved Rockstar had a very entertaining presentation again :)
He continued to make it clear that Cider, although being a Emacs extension (that Rocks!), is backed by a wealth of tooling libraries made by the Clojure community itself. One not so often mentioned library was MrAnderson, which relocates dependencies of the tooling libraries to avoid conflicts with the project using the tooling.
nREPL, the library that enables tooling for Emacs, Vim, VS Code (Calva, named after Calvados, which is distilled from Cider :)) and many more clients has a new logo and documentation website. The design of nREPL seems solid and a lot can be done with middleware (Clojurescript support via Piggieback as an example). A transit middleware seems to be in the works too.
All this community work happens with marginal funding so help in this regard would be good.
Bozhidar was skeptic that socket repl and prepl would be just re-inventing the wheel instead of making a better unified whole.
The continued effort on Clojure tooling got new powers on the second day, when Bozhidar announced that Christophe Grand would be joining the nREPL team in bringing the work explored in Unrepl to Cider (elided collections, client-side injection and sideloading for starters). This could be a direction that unifies the powers of the REPL landscape, a Hegemony as Bozhidar himself put it :)
Day 2
Keynote: Zeno and the Tar Pit
Christophe Grand explored a way to escape the Tar Pit of the internal state handling in a non-trivial stateful application with the help of Greek philosophers. He coined the term “Maps Fatigue” to describe the tension of building, converting and maintaining hierarchical in-memory data presentation in programs while coping with changing requirements.
The realization is that while Clojure lifted us from imperative programming model of Java with immutable data structures, we still have iterative processing model with seqs, while iteration order is not always meaningful (nor is the intention of iteration order easy to deduce by reading code). The base of this realization is Christophe’s past few years work on stateful stream prosessing and client/server applications.
This set us on a path on exploring databases and Datalog as a solution for the Functional Relational Programming model of the Tar Pit paper. The talk pointed to a good reading resource, recommended by Rich Hickey himself :) Foundations of Databases. He covered Datalog syntax, how to make it more readable and how to introduce a time model to be able to make updates and not only queries.
There was a teaser announcement on the final slide of a joint venture with couple of Clojurists called Tensegritics Consulting.
Applying Clojure’s Design Principles to its Documentation
During Tom McTighe’s talk, one could ask if “Clojure has good documentation” is an Oxymoron :) There’s a gap between the clarity of the language and it’s general documentation. Although there’s fair amount of material, it’s not too well organized (i.e. https://clojuredocs.org vs http://clojure-doc.org).
Klipse on the https://clojure.org/ website got mentioned in discussion after the talk and Yehonathan asked for support for persuasion to make this happen, since having a quick way to try out the language on its main website would not be a bad thing.
Quick and Easy Testing for Busy Mums (and other Clojure developers)
In her talk, Eleanor Mann presented a practical and lightweight approach to testing. Having previously worked as a Java developer in an environment that valued 100% test coverage, her experience had been that even the smallest change would break a large number of tests it would take a long time to go through them and fix them or to actually see whether there was a problem.
One of the main takeaways from the talk was that the best tests are diagnostic tools. After all, one of the reasons why we write tests is to make sure others don’t break our code in the future. This means that it should be apparent why the test exists and what it means when it’s failing. What this means in practice is that if you are, for an example, writing a test for a function that returns a map and in the test you are only interested in the value of one key, you should only check that instead of the whole map. This way it is clear for everyone reading the test what is being tested and changes to other values in the map do not break unrelated tests (presumably there would then be other tests for them).
Eleanor presented the concept of test lifecycle. Part of it is that tests should be written to be maintainable. As mentioned above, the test should communicate their intent to the reader. Test code should also be treated as proper code and instead of becoming a huge mess over time, it should be improved in the same way as the application code itself is being treated. This way it stays usable in the future.
Part of the test lifecycle is also that if the test is no longer useful it should be deleted.
The latter section of the talk was dedicated to tackling a problematic test suite. This boiled down to a few points:
- Don’t eat the whole elephant at once - you don’t need to make the whole test suite better all at once
- Don’t continue bad practices - instead for the new test code, use better practices and improve the test code that way
- Don’t be fooled by bad tests - there may be tests but that doesn’t mean they are good and that they should be kept
- Uncertain is same as no test at all - test may actually not check anything useful or it might be unstable. In this case it’s better to just remove the test and add a new one.
The talk was a good summary of a practical way to writing and maintaining tests. In itself it wasn’t particularly tied to Clojure but seemed like good advice whatever your selected technologies may be.
Using Duct Framework in Production
Iván Perdomo gave an experience report on using the Duct framework. Duct is a modular framework for building server-side applications in Clojure. They moved a report generation application from Java to Clojure and while adding components to the application, realised the need to manage dependencies between components.
The presentation nicely told that Duct is actually a reference implementation on how to use Integrant. There is a tension between a library and a tool that creates a project template with a pre-defined layout set up for a particular set of libraries, since project templates are hard to upgrade. But even though Duct has a Leiningen template for starting off, it seems to have a good upgradeability via Modules, which are pure functions that can transform the project configuration.
On the hallway track, it was mentioned that Duct might be arriving to version 1.0.0 with better documentation so stay tuned :)
Lightning Talk: Some Thoughts about Immutability and its Limits
Karl Brodowsky shared a brief look into immutability, its limits and how to work around them in Clojure. Clojure’s data structures are by default immutable, in that each mutation results in a new copy instead of modifying the original data. This seems prohibitively expensive at first, but in actuality Clojure’s data structures perform some clever data sharing under covers to avoid a full copy on each mutation and are thus quite efficient. In some rare cases however it may be necessary to squeeze out those extra milliseconds and resort to mutable data. Clojure has tooling for this in form of transients, which essentially enable turning off immutability in an isolated and performance-critical part of your code. Karl advocates keeping your APIs immutable and resorting to transients internally, only when necessary and only when you know what you’re doing: keeping the rules, i.e. accepting and returning plain immutable data structures in your API means your code stays composable with other libraries and safe for multithreadeding.
Lightning Talk: Reagent and LeafletJS or “Fun with Maps”
In this lightning talk Oliver Godby spoke about using Reagent and LeafletJS to add a map component (not the data structure, the geographical map!) to a web page. Using JavaScript libraries that don’t offer existing React components to begin with may not always be easy with Reagent and ClojureScript. In case of LeafletJS, there are existing React components that could probably be used with Reagent, but even that may have its problems, especially if you are not familiar with JavaScript and React.
The gist of using LeafletJS with Reagent came down to using form-3 Reagent components. Most of the time with Reagent it is not needed to define React lifecycle methods yourself. In the case of using non-React component with state, like LeafletJS is in this case, you need to define them for the component to be initialized and updated correctly.
Due to the fact that this was a lightning talk, we didn’t get to hear further details how the implementation went. Perhaps a little more insight what was needed with LeafletJS in specific could have been brought up.
Lightning Talk: Fullstack Clojure in the Movie Business
In his lightning talk, Nigel Runnels-Moss first described their usage of Clojure for a system for predictive analysis of the movie business (e.g. calculating optimum release times for films).
However, a major portion of the talk focused on the difficulty of hiring for Clojure development positions. As a solution, Nigel suggested to “hire for empathy’, meaning to look for competent developers that are capable of communicating with you and your team, and letting them pick up Clojure on the job.
Lightning Talk: Levelling up Lacinia with Spec
Antony Woods sneaked in Clojure in his previous company but now at Functional Works, using Clojure requires less sneaky actions :)
He presented leona, a library that generates lacinia schemas from specs. This came out of the necessity of getting rid of maintaining Lacinia schemas and Specs, both of which were a fair amount of code. spec-tools provided good leverage in implementing the library.
They also use venia to generate GraphQL queries on the client side.
Clojure Embedded in Music Hardware
Chris McCormick’s talk is probably best experienced by watching and listening the sounds produced by the hardware (try out the links to mentioned later too).
Chris is fond of Clojure and uses it out of it’s normal domain, namely in embedded hardware. There are surprisingly many Clojure -like languages and Chris maintains a list of them in awesome-clojure-likes. It turns out that code that deals with common math is shareable even throughout the dialects.
Chris also demoed a to-be Kickstarter MIDI controller programmed via Ferret language, which is a Clojure dialect that compiles to C++ that runs on the microcontroller. Another demo was running lumo on Raspberry Pi, playing a mod track. The last demo was a 8-bit sequencer that’s live-coded in the browser, which was very cool!
Growing a Product with Clojure: Lessons Learned
Łukasz Korecki told a story of progressive use of Clojure in replacing parts of a product initially written in Ruby.
He touched many topics, such as debugging performance (hint, strip whitespace to speed up the regex implementation in Java) and learning to tune the JVM (although plain copy of Riemann JVM flags sounds a bit scary :)). The big win for them was the ecosystem in Java (availability of SDKs and libraries) and also in Clojure (nrepl, ring, component etc.) and the stability of Clojure itself, which makes for a good foundation for a business.
On hiring, they were lucky to get an experienced lead developer for the backend at the start and people coming in were very interested to learn the language.
REPtiLe - A Cold Blooded Collaborative REPL
Ray McDermott of defn podcast gave a very collaborative presentation that got the whole audience involved, although the REPL REPL sing-along might have worked better after a beer (or maybe two) :).
This served as an intro to the topic of collaborative coding tools that would serve better the apropos clojure screencast, where a set of panelists talk on Clojure and code in a repl. Tmux had not worked well for them so Ray had created a tool that allows multiple users share evaluation of Clojure code from a web browser.
At the end, Ray pulled in Bozhidar and Yehonathan to use REPtiLe in a demo, which turned out a bit cold, but was entertaining anyway.
Parallel All the Way
Renzo Borgatti, the author of the upcoming Clojure, The Essential Reference ended the presentations by live-coding demo of a library called parallel, which he has been developing.
Although Clojure has built-in functions for parallel computation (pmap, pvalues, pcalls, reducers/fold, core.async’s pipeline), the parallel library seems to have many useful parallelizing functions and forms like p/let, p/slurp and p/sort.
The examples presented showed very fitting use of the library, almost “just add the alias p/ to regular code”. There’s some prior art in libraries like claypoole, which allows to limit concurrency passing a threadpool for the supported functions. Nonetheless, there’s a fair amount of new useful work offered by the parallel library.
ClojureX Park Bench Panel
The conference ended with a panel discussion with Jon Pither (JUXT), Bozhidar Batsov (toptal), Charlotte Fereday (ThoughtWorks) and Joachim Draeger (Signal Media).
One topic the discussion touched was learning resources and we were happy that the Clojure MOOC got mentioned by the audience. Enhancing this course more could be a big win :) ClojureScript could also be an angle to get more people into the language since it’s availability in the browser for creating good interactive content.
Conclusion
All in all, the ClojureX event was a great one! None of us had been in ClojureX before and we can fully recommend the event. The talks were great and so was the hallway track, which is a sign of active community. All the talks are available on the conference website, so if a talk sounded interesting, you can still go watch them.
PS. One brief closing note: The Clojurians Slack #clj-commons channel seems quite active after the conference, so if you are interested in the continued maintenance of orphaned libraries, this is a good place to follow.
