EuroClojure 2017 was held in Berlin, Germany on July 20th and 21st. The venue was Kulturbrauerei, an old brewery turned into a cultural center, in the lively Prenzlauer Berg neighborhood. This year, there were seven Clojure enthusiasts from Solita attending the conference, and a couple more from our subsidiary Evolta.
This post contains a quick recap of the talks, as well as tips on which talks you should watch from Youtube
Day 1
otplike – Erlang/OTP processes and behaviours for Clojure, Alexey Aristov
The first talk of the conference was about bringing some of the concepts and proven design patterns for building large scale systems from Erlang/OTP to Clojure with otplike library. otplike uses core.async and core.match under the hood, and if you are already familiar with Erlang/OTP patterns, you can use them in similar way. It’s easy to run into pitfalls using the low level core.async itself, so the concepts of processes, process linking, and supervision are helpful in avoiding those pitfalls.
However, not all of the features from Erlang/OTP can be implemented, due to JVM limitations. There also is work in progress for a ClojureScript version of the library.
The Return of Clojure Data Science, Elise Huard
Elise’s talk was about the current state of doing Data Science with Clojure, and proposed a roadmap for going forward.
Data Science is exploratory, so doing data manipulation in a notebook is helpful, as you can easily see how you ended up with the results. According to Huard, Gorilla-REPL seems to be the best notebook currently available. Even though the widely used Jupyter has support for a Clojure kernel, it has some issues with loading external libraries.
Clojure is a powerful language and the builtin data structures suit well for manipulating data. There are some existing libraries, like Incanter, that use their own data structures, but some work is under way to rewrite those using core.matrix. With regular Clojure data structures, you can also easily use things like clojure.spec and transducers.
Example showing data processing of an EU basic income survey
The proposed roadmap for the future included improving notebooks for easier exploratory data science, adding Clojure bindings for existing Java libraries and extracting good parts from other libraries into existing or new Clojure libraries.
Moving People with Clojure, Abhinav Sarkar, Kartik Gupta
Go Jek is an Indonesian company providing a logistics and transport service platform, where customers are placing orders and producers can fulfill those orders. Kind of like Uber, but with many other services too.
The talk was about the rewrite of their backend platform to Clojure. Originally, they started with a Java monolith that was rewritten into Go microservices in 2016. However, adding new features was slow using Go and their code was hard to read. With Clojure, they were able to make the domain heavy code easy to read and understand even by business people, for example by using function compositions and threading macros.
The Clojure rewrite also focused on testing. They wanted to duplicate their existing features exactly as they were, as that enabled them to test the implementation with live traffic and run the implementations side by side in production for a while, until they were certain that there were no issues. The talk also had some quick notes about monitoring microservices and pushing configuration updates to the live service using etcd.
It would have been interesting to hear if there were any issues with scaling their service, as their growth rate was quite huge.
The Feynman Machine: AI architecture in Clojure/Neanderthal, Louise Klodt, Fergal Byrne
This talk focused on a different approach for deep learning and AI. Instead of recursive neural networks with back propagation, the Feynman Machine uses a network of intercommunicating nonlinear dynamic systems and tries to replicate the way human brain works. This approach is good for temporal data, but on the other hand, it’s really bad for e.g. image recognition.
The authors are currently trying to reimplement their existing C++ software OgmaNeo with Clojure using wrappers and Neanderthal. They have a posted a video (and source code) of a small self-driving car running on a Raspberry Pi 3 using OgmaNeo’s implementation of predictive hierarchy.
Building a collaborative web app with ClojureScript, Gijs Stuurman
Gijs has created a very nice collaborative (think google docs) Clojure evaluation environment called CREPL. It allows two or more people to edit the same Clojure code and evaluate it live in the browser. The interesting part was an integration to Reagent atoms that keeps the atom value in sync with the text in the buffer, so even the app state can be shared between participants.
The later part of the talk was about an algorithm called Operational Transformation (OT) that is used to implement the collaborative editing, and about how to test such an algorithm in a generative way.
Testing operational transformation
Clojure Tools for Symbolic Artificial Intelligence, Saul Johnson
Symbolic artificial intelligence uses a human readable representation for problems and logic. Symbolic pattern matching, operator search and legal move generators are important concepts in this research area, and this talk presented a couple of Clojure libraries for these. With them, you can use STRIPS (Stanford Research Institute Problem Solver) style operators to specify rules and matcher functions to apply them to a set of data.
Pattern matching using matcher library
Dependency Heaven, Alex Miller
Alex Miller’s talk was about the obstacles on getting Clojure 1.9 out the door and improving first time use of Clojure for beginners. He started with a recap on Rich Hickey’s keynote from last year’s Clojure/conj; one point that appeared in the conversations to follow was that giving new names to avoid breakage is an underused solution. Also while maven itself is a great service, Clojure doesn’t really care about bytecode artifacts, since it ships with a compiler.
With the separation of Clojure core into multiple artifacts, meaning you can’t run Clojure with a simple “java -cp clojure.jar clojure.main” command anymore, there’s an avenue to rework dependency resolution :). At the same time Clojure is getting an installer, a library for dependency resolution (tools.deps.alpha) and a startup script (so you can ditch ~/bin/clj :)) which is to be published with brew, apt etc..
Tools.deps.alpha resolves dependencies from a deps.edn file, which contains a map, not a vector of dependencies. This means that ordering of dependencies to indicate desired dependency version is over, now taken care by tools.deps.alpha. The clj script then reads deps.edn and so could existing tools too. It remains to be seen how this will take on in tooling, but there definitely is a place for fresh take on dependency management here.
Save the Princess, Simon Belak
Simon Belak’s talk showcased an example on how to build an AI for a simple “game” where you have to find the shortest path to the princess (that you need to save) from a random starting location in an XY-space. The approach was again slightly different from the more popular back propagating deep neural network trained using reinforcement learning. This time, the AI used evolution strategy which seems to be highly parallelizable. It was first trained with a policy network using Cortex, then modelled using Neanderthal and finally the model was also parallelized using Onyx.
It would have been nice to see how the model actually performed, and how long did it take to train it. Nevertheless, Simon covered many systems, although a thorough dive into one subject would have brought more focus.
Unsessions
After the first day, there were a couple of unofficial “unsessions” around the venue where a group of enthusiasts gathered to talk about a predefined topic. These topics included e.g. core.spec, the future of REPLs, making Clojure experience easier for beginners and Clojure Data Science.
Day 2
Intro to Probabilistic Programming and Clojure’s Anglican, Nils Blum-Oeste
Probabilistic programming can be used to predict the future, infer causes of observations from incomplete data and for improving models from evidence. Nils provided a recap of Bayesian statistical models and theorems, which can be used for improving predictions of probabilities by taking a prior distribution and producing a posterior distribution based on evidence.
Nils showed a couple of examples using Anglican, which is a Clojure library for probabilistic programming. Anglican makes an interesting choice of providing a language that is a subset of Clojure. Programs are written in the body of defquery macro, in which some symbols (like map) collide with Clojure core, which might look a bit confusing at first glance but the intention is to provide a language suitable for presenting for example observations from a probability distribution. It’s great for modelling and prototyping, but inference is slow and the ecosystem is lacking.
Building a probabilistic model using Anglican
spec loves data structures, Michał Marczyk
Michał really is the reigning data-structure library guru in Clojure :). He showed how he does testing on data.avl with clojure.spec. The way he can test code that uses deftypes with spec is by doing transformations between the types and map based representations.
He also showed two interesting bugs that have been caught be generative testing (collection-check): one in Clojure’s own PersistentHashMap and another in data.avl. The bug in data.avl prompted him to start doing white box testing of the internals of his libraries.
My conclusion after seeing the sample test code is that I can trust any of Michał’s libraries in production despite their low version numbers… and so can you.
Simple AND Secure, Joy Clark
Joy Clark (who is also known for her excellent sketch notes from the EuroClojure presentations) talked about how to make secure Clojure web applications.
Continuing the Hickyan tradition of simple made easy, making secure web applications is not easy :). There is no magic framework for security, especially in the world wide web of today, so one has to keep a good hold of best practices. So how to write a secure web application? Maintain, stay informed, KISS, know what you are doing and monitor your application.
Joy also presented examples of OWASP Top 10 issues and how to avoid them in Clojure. The talk was illustrative and easy to follow, so if you are not familiar with web security basics and how to make web applications secure using Clojure, you should definitely watch this talk.
With Ring, you can easily combine many small parts into a single system
Tear down this wall - rethink distributed systems, Christian Betz
Christian Betz had a very fitting title with the conference venue :) Think about distributed systems as a whole instead of siloing them into separate teams. “Get rid of borders but keep border control” was neat way of saying API design matters. Spec for incoming, outgoing messages and application state was a good reminder. Consider using cljc to be able to run tests on both the JVM and on JS, like re-frame does. Availability of Clojure in all parts of the distributed system leads to a common way of thinking, since language used determines how we think.
The subject of the talk was quite broad and possibly paved by personal experience. The subject of monorepos was also mentioned, lein-monolith being one tool for taking every downstream project to create a repl.
Using Clojurescript to launch iOS/Android apps to 1M users, Emin Hasanov
It’s always interesting to hear experience reports, and this talk told the story of Tap.az, an Azerbaijani website for classified ads (the local Craigslist), and their mobile application. Even though their regular website was mobile friendly, there was still a demand for a native application. They wanted to launch and iterate quickly, use the same codebase for both platforms and avoid having to use the native languages for the platforms. The talk described a couple of options ranging from a simple website wrapper to “almost native”. In the end, they decided to use React Native and since they had a developer with Clojure experience, used reagent and re-natal instead of the vanilla JavaScript React Native.
An interesting point was that even though they developed the first version of the app in only three months, they spent over two months in just fine tuning the UX and small details. Pushing updates using app stores is also really slow, but with services like CodePush, you can do small changes much more quickly.
A year after launch they have over 200k installations and over 20% of their daily sessions are coming from the app.
The happy conclusion here is that React Native with ClojureScript is ready for production (for the brave).
Unravel your REPL, Paulus Esterhazy
What is it like to program in Clojure?
Clojure has a different approach from static languages like Java. The Clojure approach is to break tasks into parts, test each function with sample input data and repeat until results match expectations.
Phenomenology of Clojure
Debugging also has a bottom-up approach as well. The difference in approach is what makes Clojure programming feel different to static languages. That difference is the REPL experience which is at the core of Clojure (and lisps in general). When you primarily work in a REPL, you need good tools for it.
Paulus talked about two projects related to Clojure REPLs:
- unravel is a new terminal REPL with nice features
- unrepl is a EDN based REPL protocol
Unrepl neatly separates prompt and result values in their own messages, output is not just a character stream with prompts, printed output and results intermingled. Can upgrade a normal clojure REPL into an unrepl session by sending the unrepl host code to the server as bootstrapping and waits for the server to respond with an edn value.
Definitely an interesting talk even if you are happy with CIDER. Perhaps a new ecosystem of REPL tooling will bring about more experimentation on novel features.
Microservices with Vase and Linkerd, Peter Brachwitz
Microservices are a good fit for Conways law which states that organizations produce copies of their own communication patterns in the systems they build. But the interesting promise of microservices is that you can have teams that work with different tools and at a different pace.
Peter described Clojure tools for creating microservices in Clojure with production quality. Vase is a framework for data-driven microservices. Vase provides an HTTP API for Datomic based services. The HTTP API is defined as Pedestal routes.
Another layer of microservices is how to link them together and make the communication between them robust and tolerant of errors. That requires a service mesh or service proxy like linkerd or envoy.
Linkerd is a transparent HTTP proxy that routes traffic between services. Linkerd takes care of the difficult details when calling services: handling errors and retries. Linkerd can also do traffic shifting by routing a percentage of calls to another version of a service.
Linkerd also has a service called namerd that handles routing for it. It is unclear to us why it is necessary and why would you not just use DNS.
The main point about this talk probably is that all the complexity here is incidental complexity. None of it actually has to do with the problem domain, but it is required if you want to destructure the system in this way.
Generative Programming and Verification, Nada Amin
Live coding, yay! (but in Scala :()
Nada showed her research projects about writing programs that write programs. Sound like lisp macros, right? Well, not exactly. The system is called LMS (Lightweight Modular Staging) which provides a way to create embedded compilers in Scala. The Scala code has types for representations that split the world into the compile and run stages.
Nada showed us a live coding example of doing loop unrolling in matrix operations. The approach is not limited to generating Scala code but can be used to generate anything (for example C code).
Loop unrolling
From a lispers point of view, code generation is our bread and butter and the Scala code seemed to be sprinkled with type declarations everywhere. It’s nice to see that Scala can have macros as well and it seemed easy to use.
Conclusion
Many things in Clojure are being done in many ways, from starting up Clojure to firing a REPL to running a service. There’s a lot of active work on the tooling of Clojure.
The best part was meeting other Clojure enthusiasts and seeing the vibrant Clojure community. See you next year!
