Content tagged EN

Learn You a Haskell for Great Good

posted on

This is the first book on Haskell, that really made me want to read it. I had it with me as an e-book on my vacations. I couldn't stop reading it before I finished. It is a very good introduction to the basics of the Haskell programming language. It even contains an understandable explanation of the infamous monads. (Side note: they are a very simple concept, and you shouldn't fear them.)

For me this book was the right amount of introduction to get me started. Everything else I think I will be able to learn while using the language.

The book is available in print at your preferred book shop, or can be read online and free of charge.

Links to resources

Getting Safari Books Online

posted on

I like the books from O'Reilly. For many years I've seen the ad on the backside advertising Safari. Sure, I checked out what this is, but I wasn't really interested to have a digital copy for a limited amount of time. I already had the print version.

I love to read real books made of paper. I use highlighters and add notes, while reading. I make the book mine. And after finishing the book, I can place it on my bookshelf, to show the books I like to others. This doesn't work with my Tolino e-book reader. While I have this device, I still buy most books as a paper copy.

Some days ago, I heard of Safari again on a podcast. I thought, that at least I should check it out. It's not only O'Reilly books anymore and it includes videos and other podcasts as well. There is a free 10 days trial, so I did not have to invest more than some of my time.

It didn't take long and I was captivated by Safari. I noticed how cool it is to checkout nearly any computer book I am normally reading. Curious about something? I can just log in and start reading. It didn't take long, that I had a regular, payed subscription …

The price? It's quite expensive. I have to pay around 40 € per month (including VAT). I saw several people on the net comparing this to about two e-books a month. This comparison doesn't work for me: I still want to read paper books. For me Safari is a tool to fossick for new content.

Another big down-side of Safari is, that you have to read it on the web, or with an Android or iOS app. This limits the user to read on devices with a screen not optimized for reading. You cannot read using an e-book reader with an e-ink display and much less weight per screen size.

For the moment I am still enjoying the service as a big index to books I might been interested in. To bad, that there is no offer for people using it and buying the books at the same time. As a Safari subscriber I'd like to get a discount on the books I buy. Or they could offer some free days for every bought paper book.

Link

Reactive Programming with RxJava

posted on

RxJava is an implementation of “Reactive Extensions” (Rx) in Java. But what is this? Originally Rx are an implementation of Microsoft to address the increasing complexity of software. It is a model to build large scale asynchronous service architectures.

Another company that has to be mentioned is Netflix: they implemented the Rx in Java which resulted in RxJava.

On the programming side Rx looks very similar to how streams (java.util.stream.Stream) work. The main difference in my opinion is that streams are pull and lazy. They produce only as much data as someone is reading from them. RxJava on the other side is more push style. The source of data is pushing events through a pipe of operators similar to what you know from streams (filter(), map(), and so on) to the sink. And as I know from this book, Java streams are the wrong tool to parallelize anything that is not CPU bound–like network requests. There reason for this is, that (parallel) streams are executed on a thread pool that is shared with several other features of Java. This thread pools is limited to have only that many workers as the system has CPU cores. Therefore this pool gets exhausted very soon, when threads in it get blocked by I/O operations. Any thread waiting for an I/O operation effectively results in a processor core not used by your program.

RxJava on the other hand is not limited to a fixed thread pool. Any source (Observable) and sink (Subscriber) of data can be bound declaratively to user defined Schedulers. As well as Rx favours a model in which I/O operations are done asynchronously and non-blocking. Therefore resulting in a need for much fewer threads. In a traditional model of using one thread per network connection, threads become very soon the first thing that limits scalability.

What is really great about this book

The best part of this book for me were the reflections on Relational Database Access in chapter 5. While as a developer you might be tempted to convert everything to the reactive model, this part of the book shows where it doesn't make sense to do so.

By converting the access to your relational database to an asynchronous model you won't gain anything. Whatever you are doing on the client side, let's say for example your PostgreSQL will run all of your concurrent requests in different processes. This results in a noticeable limit on the number of parallel queries you're able to run. You cannot lift this limit by becoming asynchronous on the client side.

Links to the book

ZooKeeper, Distributed Process Coordination

posted on

ZooKeeper is a component that facilitates building distributed applications. It is:

The data managed by ZooKeeper is presented in a file system like manner with directories and files whose names get separated by slashes (/). The difference to a file system is, that you can store information in the directories as well. Or seen differently: directories are files at the same time. Based on this simple abstraction, users of ZooKeeper can implement things like leader election in a cluster of software instances.

The book by Flavio Junqueira and Benjamin Reed

The book is written by two experts of ZooKeeper, that know how it works internally and what are the pitfalls in which the users can trap. Flavio Junqueria is one of the ZooKeeper's contributors. Benjamin Reed helped to start ZooKeeper.

I was reading the book, because I is the basis for other distributed software systems I made myself familiar with the last months, including Akka and Mesos. I always think, that it's a good idea to know at least one layer below the layer I am actually using. I allows me to understand better what I'm doing and how to do it right.

The book starts by giving an overview of the concepts and basics used by ZooKeeper. It introduces an example master-worker application, that is implemented using different languages afterwards:

Other topics discussed in the book are:

I think, that this book is a highly valuable resource for anybody working with ZooKeeper either directly or indirectly by using some other software, that uses it.

Links to the book

Microservice Architecture, anligning principles, practices, and culture

posted on

To stay on the right track with microservices, I wanted to revisit the philosophy and organizational recomendations on how to do them right. After reading Building Microservices in april this year, I got Microservice Architecture, aligning principles, practices, and culture by Irakli Nadareishvili et al.; O'Reilly Media, Inc., 2016.

The book can be read on one week-end as the content is very well condensed to 118 pages.

What did I get from the book?

The book provides a lot of information on the culture and organizational requirements to successfully implement a microservice architecture.

[C]ulture is perhaps the most intangible [domain] yet may also be the most important. We can broadly define culture as a set of values, briefs, or ideals that are shared by all of the workers.

Microservice Architecture, page 29

It was very interesting to me to compare the presented observations with how we are doing microservices at work. One thing was especially interesting for me, as I did not read of this before, was the mention of the paper The Social Brain Hypothesis, 1998 in which Robin Dunbar writes about observations that there are several sizes of groups, that work better then others.

[T]he various human groups that can be identified in any society seem to cluster rather tightly around a series of values (5, 12, 35, 150, 500, and 2,000).

The Social Brain Hypothesis, Robin I. M. Dunbar, 1998

I can definetly recommend this book to everyone that already does microservices, or that plans to implement them. It does not present nor recommend any specific software product, but talks about the social and organizational requirements to succeed with this type of architecture. And it notes that they are not always the right way to go. In the book there is also a detailed list of further sources to deepen on various aspects of the topic.

Links to the book

Developing Web Components, UI from JQuery to Polymer

posted on

It's hard to find sources how to do front-end micro-services in a single page application (SPA). Having a single front-end that faces the user makes it hard to impossible to exploid the full power of going micro-services in the back-end. For every new function you cannot just deploy the corresponding service, but you have the dependency to update and redeploy the service as well.

So I was looking around how to go micro in an SPA. One of the ideas I found on the web was to do so using web components. To evaluate this idea as someone working mainly on the backend I thought I should get some literature and bought the book Developing Web Components by Jarrod Overson and Jason Strimpel, O'Reilly Media, Inc., 2015.

What did I get from the book?

The end-to-end theme of the book is to develop a dialog box web component. As a familar entry to the component it starts doing so in JQuery. Afterwards the component gets ported to using plain HTML5 APIs (templates, shadow DOM, custom elements, and imports). In the last part of the book, it gets converted to using Polymer. This makes the implementation much cleaner and easier to do.

As I don't do much front-end development it was very interesting to see what is possible in modern browsers by now. I like how web components bring back some of the strengts of HTML: marking the structure and semantics of a text instead of building interfaces by just styling a bunch of <div/>s. I like to see hightly semantic pages on the web. But I see so many web developers only taking care of the visual impression in a browser. Therefore it's always a requirement, that the way to get there also results in the work being done easier. The ease of using components built with polymer may achieve this goal.

But pertaining to my initial motivation to read the book, I have to say, that I didn't find a good match for doing micro-services in using web components.

Recommended links

Locked-in by Google

posted on

For several years now WhatsApp is spreading on the smart phones of my contacts. Of course my friends are asking me to install the software as well. But I hate using a proprietary service if there are open alternatives. This open and comprehensive alternative was and is Jabber respectively XMPP (two names for the same thing).

But this is nothing more then a protocol. Especially nothing anybody could use. What a Jabber user needs is software and a service provider. Until now my recommendation for both was Google Talk. The software perfectly integrated and pre-installed on every Android phone. An easy to use user interface, comprehensible to John Doe. Nothing made for freaks by nerds. And as an operator Google provided a rock solid service.

That was a smasher: you could get everything well co-ordinated from Google, you could chat with users of other providers, and you could even operate our own server if you wanted to. Real class!

With my denial of using a closed system and my continuing referral to free alternatives, I brought several contacts to Google Talk.

Google Talk gets Hangouts

At the I/O 2013­–Google's annual developers conference–"Talk" has become "Hangouts". A reasonable move. The last years Google had started several communication products, that had not work well with each other. Besides "Google Talk" there was "Google Voice", "Google+ Messenger", the previous "Hangouts", and some time they even had "Google Wave". For a user it was confusing to have different products to do essentially the same: communicate with their contacts. You could have your contacts multiple times: in Talk, in Google+, and so on. Before you started to chat, you had to decide which application to use this time. Thus I am completely fine to integrate everything into one product. Using a communications product must be easy and clear.

But actually there was another thing that happened at the same time: Google decided to close their platform. Until now Google users where able to communicate with users of other providers. Nikhyl Singhal, director product management real time communications at Google expressed it (starting 5:07 in the video below): "Hangouts is not based on top of the XMPP standard. […] We have essentially made a hard decision to focus less on the XMPP standard and more on what are users looking for." Well maybe that sounds good, but with regards to content it is dreck.

As I said XMPP is a protocol and nothing the user is working with. Users care about the experience, the interface, and the fun. Where is the conflict? Why not concentrate on all the things a user cares and still building this based on top of XMPP? XMPP is the "eXtensible Messaging and Presence Protocol". The key word here is "extensible". There is really no problem to create all the nice things Google wants to deliver using XMPP.

In my opinion the actual reason of Google to close their platform is something else: Google wants to look users onto their services. It is not wanted anymore, that users may communicate with users of other providers. Google wants these other people to have an account at Google as well.

This is very similar to another shut down of a Google service next month: Google Reader. This as well is a product, that allows Google users to subscribe to and read feeds from third party services. Google closes this service supposedly in the hope, that former Reader users start using Google+ instead. (No need to mention, that on Google+ you can only subscribe to content of other Google+ users.)

What now?

The reason why I used Talk instead of other messengers has fallen away. I do not want to be caged in the service of a single provider. My formally Google hosted Jabber address m@tthias.eu is now hosted on a free server again. I am stilly also reachable using the same address on the Google Hangouts platform. But probably this will stop sooner or later.Until then I am looking for a user friendly Jabber application for Android, that I able to work well with mobile internet connections. Who knows, maybe the application is already there, I just do not know it yet. After Google announced the shut down of Reader I also was surprised to find Feedly. Some software that already did exist and feels better than what I used before. Recommendations are always welcome.

And what's about WhatsApp? If Google is not better anymore, will I start supporting what "everyone else already uses"? No, I do not think so.

Further reading

History of Matthias Wimmer's PGP and GPG keys

posted on

PGP-signed version of this history. - Text version of this history with a qualified signature.

KeyID Fingerprint Date Expires (sign/encrypt) Status
70D6C898 CAEC A12D CE23 37A6 6DFD 17B0 7AC7 631D 70D6 C898 2006-08-10 2010-01-01/2010-01-01 This is my main GPG key since 2006-08-10.
C9CD24F7 85A0 801A 2392 424F 69AB 5A63 C6D6 197D C9CD 24F7 2003-03-06 2008-03-04/2008-03-04 Key in use for special purposes. Please don't use it for e-mail.
4E59C7E6 B0F5 9D8C 28A8 34B6 FDE1 410A 3D27 3DDA 4E59 C7E6 2003-01-21 2007-01-01/2006-01-01 Key still exists, but only used for migration-signing to new key anymore.
8D8B4A2E 6F81 B414 B8A1 1806 A333 A18E 0142 F366 8D8B 4A2E 2001-01-11 *** revoked *** Public and private key still exist, never used for e-mail/not used anymore
AA839AF9 85E8 0EE5 C852 363C FC19 BD5F 27FE 6356 AA83 9AF9 2000-03-06 *** revoked *** Public and private key still exist, but not used anymore.
034FDE2A EA16 DEA8 4146 12EE 3B56 68D8 0A70 794E 034F DE2A 1999-09-15 *** revoked *** Public and private key still exist.
55DB8129 AF 14 32 B5 69 38 30 6E 3A B8 13 8D 66 A8 66 AE 1999-04-27 never :-( Lost private key!
B3D1AF25 8C 79 81 AA ED 95 4D 0B 8F 53 09 52 9A 39 32 49 1998-05-25 *** revoked *** Public and private key still exist, revoked because e-mail address is not used anymore.
730BD791 ??? 1997-07-05 never Lost public key, private key still exists.
D9954A11 ??? 1996-10-23 never Lost public key, private key still exists.
3336E4E9 ??? 1996-07-24 never Shared key (for the sysops of my former mailbox), I am not the only one that has the private key. Public key lost.
8B674F75 DE 51 C1 7E E4 99 4B 9D 5B 76 06 B2 DF 00 64 F1 1996-07-18 *** revoked *** Public and private key still exist, revoked because e-mail address is not used anymore.

There exist some even older keys that I have never used in the internet and that only contain FidoNet addresses. These keys are completely lost. I neither have the public nor the private keys anymore.

Change history

2006-08-10: Marked key AA839AF9 as revoked, updated signing-expire-date for key 4E59C7E6, added new key 70D6C898
2003-12-04: Updated expiry information for 4e59c7e6

IMUnified - protocol

posted on

In 2000, IM Unified announced to create a protocol that enables an instant messaging client to join different IM services. Founding members were MSN, Odigo, Yahoo! and others.

Since then I have not heard much of IMU. I don't even think that their IMIP protocol will ever be used really. But it does exist and e.g. the Yahoo! messenger client supports it. You only have to change some values in the Windows registry. Afterwards, you're client is able to join other IM services like Odigo with the Y! messenger.

For no other reason than to see how IMIP works I've dumped some IMIP sessions with a network sniffer. The IMIP protocol is very simple structured and looks a bit like SMTP or http.

All data is exchanged over a TCP/IP connection. The clients connects to port 11319 at the server. Firstly, it performs the login handshake and afterswards, it is exchanging messages, presences and other data. All data is exchanged in blocks.

All blocks have a structure of lines. Every line is finished by \r\n (carriage return, line feed). The first line contains the type of a block, the second line contains the size of the block (first two lines not counted). It is a decimal ASCII number containing the size in bytes. Starting at the third line, there are head lines, optionally followed by an empty line, optionally followed by a block body. If the body is empty, the empty line can be absent. (The Y! messenger is allways sending the empty line, the Odigo server isn't.)

As block types (content of the first line) I have noticed the following so far: "HELO" (before login, version check, negotiation of auth type), "LOGN odigo-number" (exchange of login data), "STAT presence" (exchange of presence information, presence is one of ONLINE, AWAY, BUSY, OFFLINE (other side going offline or this side going invisible), "ACK status" acknowledge (status 600 is OK, I got 811 after sending wrong login data), "LIST ADD odigo-number" (adding user to list), "PING" (keep-alive packet, interval set by server in HELO block), "MESG" (message), "LIST ACCEPT odigo-number" (accept being added by other side), "DISC" (logout).

The header lines always have the form "identifier: content\r\n". Every block has an ID line ("ID: idvalue"). This ID is mirrored in a reply by the "Reference: idvalue" header line, sent by the server. The Y! messenger starts to generate the IDs at "1001" and increments this value for every block. The Y! messenger is always sending the ID lines as the first header line. I don't think this is neccessary because the Odigo server is not always sending this line as first header line.

The first block that is transmitted from the client to the server is a "HELO" block. Besides the ID line it contains a "Protocol: IMIP/1.0" line. The Odigo server replies with another "HELO" block containing the following lines: "Auth-Type: imip-md5", "Capabilities: server-lists", the ID line, "Keep-Alive: 60", "Protocol: IMIP/1.0", "Registration http://www.odigo.com/user", "Service: odigo.com", "ServiceDisplayName: Odigo", an empty line and an ASCII string (Odigo uses the decimal ASCII representation of a 32 bit value), that is used as salt for the MD5 password exchange. The Keep-Alive line seems to control the interval of the client sending "PING" blocks. The HELO block does not contain a Reference line.

Afterwards, the client sends a "LOGIN odigo-number" block containing the following header lines: ID line, "Client: Yahoo! IMIP Client", "Auth-Type: imip-md5", an empty line and the MD5 hash of a string containing the salt concatenated with the user's password. The hash is represented as a hexadecimal ASCII value with small letters. At the UNIX command line you can reproduce the hash value with md5sum. e.g. if the salt in the HELO block was "1919833824" and your password is "password" you get the hash with: printf "1919833824password" | md5sum (the result should be c35900c24a82592dd2d0db27c647ff17). The server replies with a "LOGN odigo-number" block containing only ID and Reference header lines.

After login, the client sets its presence with a "STAT status" block (status being ONLINE, BUSY, AWAY or OFFLINE). This block only contains an ID header line. The server replies with an "ACK 600" block containing ID and reference header lines. The presence can be explained additionally by a message in the body of the block.

The server informs the client about the presence of the user's contacts with "STAT status" blocks as well. These blocks contain an ID header line and a "From: odigo-number" header line. The client doesn't reply these blocks.

Every time the interval set by the HELO block expires the client sends a "PING" block to the server. This block only contains the ID header line. The server replies with "ACK 600".

A message is sent by a "MESG" block that contains the following header lines: ID, "To: odigo-number", "ACK-Type: errors-only", "From: own-odigo-number". After an empty line, there is the body of the message. The server doesn't react on this, probably controlled by the ACK-Type line.

A message is also received by a "MESG" block. The server sends some more header lines: "Content-Type: text/plain;charset=utf-8", "From: odigo-number \"nickname\"", ID line, "Timo: yyyy-mm-ddThh:mm:ssZ" (time in UTC), "To: own-odigo-number", an empty line and afterwards the body with the message in it. The client doesn't react on this block.

The clients starts a disconnect with a "DISC" block containing only an ID header line. The server replies with an "ACK 600" block. Afterwards the connection is closed.

To add a contact to the list, the client sends a "LIST ADD odigo-number" block. The contained header lines are: ID, "List: Buddy", "From: odigo-number". The body can contain a message. The server replies immediately with an "ACK 600" block containing ID and Reference header lines. After the contact has accepted the subscription, the server sends a "STAT status" block.

If somebody else wants to add you to his list, the server sends a "LIST ADD" block without an Odigo number in the first line. This block contains the following header lines: "From: odigo-number \"nick\"", ID, "List: Buddy" (I had this user in the Buddy group). This block can contain a message in the body. The subscription is accepted by a "LIST ACCEPT odigo-number \"nick\"" block, that contains an ID and a From header line. It does not contain a Reference header line. The server replies with an "ACK 600" block.

Update: In the meantime I beleave that "List: Buddy" is no contact list group but just the contact list and that there could be other lists like a invisible list. I havn't done more research on this topic yet. New is what I found out how a contact is removed (block of type "LIST REMOVE odigo-number" with the headers ID, "List: Buddy" and "From: odigo-number"; acknowledged with A "ACK 600" block by the server) and how to reject a subscription (block of type "LIST REJECT odigo-number" with the headers ID and "From: odigo-number").

In the LOGN block of the server there is also the list of buddies on the contact-list. This in the Buddy header. The content of this header is a list of comma separated contacts (with their nick names appended in quotes).

I got the following ACK status numbers besides "ACK 600": 800 after I sent a STAT block without a status, 801 after I sent a STAT block with a non existant status, 810 if I try to log in with a wrong password, 811 if I try to log in with an illegal user id or send a message to an illegal user id (the id was non-numerical, Odigo uses numerical IDs).

Simple IMIP session as an example, IMIP session with message exchange and subscription, simple Java-Odigo-Client implementing the IMIP protocol.


Unless otherwise credited all material Creative Commons License by Matthias Wimmer