DAT: A Better Way to Build a Social Network

Tags

Beaker, blake, DAT, Ed25519, Electron, js, node.js, p2p, protocol

I’ve hinted, in last week’s post, about the development of a feature that could help Minds.com evolve into a decentralised social network and bring P2P into the mainstream, thereby solving the growing privacy and censorship concerns that are associated with a centralised social network. This feature is based on an application layer protocol known as ‘DAT’. There are reasons to believe it’s likely to succeed where previous ideas failed: Since DAT works entirely at the application layer, and is implemented using Node.js, there’s very little effort or learning curve involved for developers and users of DAT applications. Web applications can be extended to support it, if the demand is there, using already published libraries that are extensively documented.
For those who aren’t developers, there is a working browser that anyone, without technical skills or knowledge, can use to browse and publish sites on the DAT Web.

What is DAT?

DAT started life with the scientific community, which had a need for a more effective method of distributing, tracking and versioning data. In the conventional Web, data objects are moved, Web pages are deleted and domains expire – this is referred to as ‘content drift’. We’ve all come across an example of this in the form of ‘dead links’. When using a hyperlink to reference a data object or Web page, there is no guarantee that link would be valid at some point in the future. DAT was proposed as a solution to this.
But what does this have to do with censorship and privacy, you’re probably asking? The answer to this question is in how data is distrubuted, discovered and encrypted.

Merkle Trees, Hashing Algorithms and Public Key Encryption

The DAT protocol is essentially a real-world implementation of the Merkle Tree data structure, with the BLAKE2b and Ed25519 algorithms for identification, encryption and verification (other docs state that SHA256 is used as the hashing algorithm). It’s not necessary to understand this concept in order to develop DAT applications, since there are already libraries for implementing this, but for the curious, I reccommend reading Tara Vancil’s explanation first before moving on to the whitepaper.

An important point Vancil made was DAT is about the addressing and discovery of data objects, not the addressing of servers hosting those objects. Data objects are not bound to IP addresses or domains either. Each data object has its own address, and that address is determined by its cryptographic hash value – a file’s hash digest will be static, regardless of where it’s hosted. This is important, because we’re accustomed to thinking of the Internet/Web in terms of the client/server model, and proposed solutions for privacy and anti-censorship typically try to deal with the problem of decentralised host discovery in a peer-to-peer (P2P) network.

Some form of data structure is required to make the data objects addressable and to enable their integrity to be verified. A DAT peer-to-peer network uses Merkle Trees for this, where all ‘leaves’ and nodes contain the hash values of the data objects they represent, and the root node contains the hash digest of all its child nodes. In other words, as the whitepaper puts it, ‘each non-leaf node is the hash of all child nodes‘.
Not only does this provide a way of verifying the integrity of the data objects – the root node’s digest will change if there’s any modification to a data object represented in the tree – it provides the means to an efficient lookup system, as the root hash digest becomes the identifier for a dataset.

Obviously, this means clients would need to fetch the root node’s value for a given dataset from a trusted source, which might be one of many designated lookup peers on the network. If the client wanted a given data object, it wouldn’t need to fetch everything referenced under the root node, but just the root node value, the parent node of the requested objects, and the hash values of the other parent nodes.

Addressing, References and Security

Now, let’s get into the more specific aspects of how Merkle Trees are implemented in the context of DAT. All the ‘leaf’ nodes in the DAT Merkle Tree contain a BLAKE2b or SHA256 (depending on the docs being read) hash digest of the referenced object. All parent nodes contain the hash digest and a cryptographic signature. The signature is generated by creating Ed25519 keys for each parent node and using them to sign the hash digest.

When sharing a locally-created site in the Beaker browser, or viewing one already shared on the network, you might notice the URI following ‘DAT://’ is a long hexadecimal string. This is actually the Ed25519 public key of the archive containing the referenced object being shared, and it’s used to encrypt and decrypt the content. The corresponding private key is required to write changes to the DAT archive.
The public key is, in turn, hashed to generate a discovery key, which is used to find the data objects. This ensures no third-party can determine the public key of a private data object that hasn’t been publicly shared.

Beaker

The Beaker browser looks very much like the standard Firefox browser on the surface, and it can be used to browse both DAT:// and HTTP:// addresses. As we can see, DAT sites are rendered just as well as those on the conventional Web. The only problem is that, as with Tor and I2P, sites are hosted on machines that aren’t online 24/7, so many of them are unreachable at a given time.

From the Welcome dialogue, we can get straight to setting up a personal Web site dor publishing on the DAT Web. A default index page, script.js and styles.css are included ready for us to customise. In addition, Beaker allows us to share the contents of an arbitrary directory on the machine it’s running on.

Previously-created sites are available under the ‘Library‘ tab in the main menu. Sites that aleady exist will be listed under the ‘Your archives‘ section, and can be modified and/or published.

What happens to a published site when the local machine is offline? There is a method to keep a site accessible, by somehow getting another person or machine to ‘seed’ the data. This is a short-hand way of saying another person could fetch a copy of the site and re-share it over the network. Seeding happens automatically as a user is actively browsing a DAT site.

The Node.js Modules

Several Node.js modules provide libraries that developers can use to implement DAT features in their applications.

• hypercore: A component for creating and appending feeds, and verifying the integrity of data objects. The API exposes a number of methods under the ‘feed’ namespace for reading, writing and querying feeds.
• hyperdrive: This is a distributed filesystem for P2P. One of the design principles is to reproduce, as closely as possible, the APIs as the core Node.js filesystem component, thereby making it transparent to application developers. This module enables a local file system to be replicated on other machines.
• dat-node: A high-level component that developers could use to bring together other DAT modules and build DAT-capable applications.
• hyperdiscovery: Module for network discovery and joining. Running two instances of a hyperdiscovery module will result in a given archive key being replicated.
• dat-storage: The DAT storage provider. Used for storing secret keys, among other things, using the hyperdrive filesystem.

In conjunction with Electron.js and Node.js, the above modules can be used to develop a DAT-enabled desktop application, of which Beaker is just one example.

Node Discovery in Practice

Two components are used for this: discovery-channel and discovery-swarm. The discovery-channel component searches BitTorrent, DNS and Multicast DNS servers for peers, and advertises the address/port of the local node. Therefore, it is based on the bittorrent-dht and dns-discovery modules. Using discovery-channel, the client can join channels, terminate sessions, call handlers on session initiation and fetch a list of relevant channels. The network-swarm module uses discovery-channel to connect with DAT peers and control the session.

Advertisements

Minds.Com and the Free Thought Project Interview

Tags

censorship, facebook, FreeThought Project, Minds.Com, Ottman, privacy

Recently I was listening to the Free Thought Project’s interview with Bill Ottman, the CEO of Minds.com, and thought it worth expanding on some of the points. If you haven’t already done so, Minds.com is worth checking out, if you’re a content creator, blogger or citizen journalist looking for an alternative to the mainstream platforms.

The Free Thought Project was one of 810 accounts that got booted off FaceBook and Twitter for ‘inauthentic activity’, in what seemed more like a co-ordinated act of political censorship. While the full list hadn’t been released, the main targets appeared to have been groups reporting on corruption within politics and law enforcement – you know, things we have a civic duty to discuss on the Web.
Quoting Brittany Hunter in Foundation for Economic Education article: ‘What began with the ban of Alex Jones last summer has since escalated to include the expulsion of hundreds of additional pages, each political in nature. […] one thing is absolutely certain: we need more market competition in the realm of social media.‘
What’s particularly worrying is that the Silicon Valley corporations aren’t simply private entities excercising their own rights, as is commonly argued in their defence. They represent a giant oligopoli that has a disproportionate amount of control over the means of communication on the Web, an oligopoli that’s engaged in a co-ordinated suppression of political opinion, an oligopoli with more influence on the democratic system and access to politicians than the Russian state could ever hope to gain.

An alternative is needed to democratise social media. For many people in the know, Minds.com seems to be that alternative. Here’s why:

• Minds is production-quality, can be deployed as a finished application, and it’s open source.
• Users don’t need to provide personal or identifying information when registering an account.
• Minds was developed for content creators.
• The developers are working on decentralisation solutions.
• Minds.com supports crypto currency and monetisation.

The first point is an interesting one. In Ottman’s opinion, a solution released as proprietary software cannot be a viable alternative, because of transparency or somesuch. I think he might have conflated administrative integrity with software integrity – that open source projects have been pressured into adopting a uniform ‘Code of Conduct’ demonstrates the problem with that reasoning. Personally I don’t think the open/proprietary thing has much bearing on a platform’s viability as an alternative to FaceBook, unless there’s a need to verify claims about certain features, such as whether true end-to-end encryption is being provided.
No, what’s more important is that Minds isn’t a half-baked proof-of-concept, but is a completed iteration comparable in quality and appearance to any mainstream social media site. This is the deciding factor that determines whether a solution would gain traction. Anyone could clone the software, deploy it on their own server and run their own version of Minds.com.

The option to register accounts anonymously/pseudonymously with Minds.com is probably the most important feature, because I strongly believe we should be setting boundaries between our online and offline lives, and between family, social circle, work colleagues and strangers. Such a thing isn’t really possible on a social network in which everyone’s posting under their real names. Also, I don’t think it’s possible, in our current political climate, to have any meaningful debate without pseudonymity, since it seems fashionable to ensure anyone expressing a dissenting opinion suffers disproportionate ‘social consequences’.

An undersold feature of Minds.com is the ease with which a citizen journalist, blogger, whistleblower, etc. can create and publish content. For the individual user, who wants to protect his/her identity, a Minds.com channel (with publicly-viewable blog posts) is cheaper and easier to maintain than a Web site, and it still provides the same benefits in terms of posting content and getting views.

Problems with the Design and Architecture

Now, for the things I’m not entirely sure about: My main criticism is that Minds.com is not (yet!) actually ‘engineered for freedom of speech, transparency and privacy’ in any tangible sense, as it’s still a centralised service hosted on AWS in the United States. Whether Minds.com defends its principles actually depends on the people running it – people who could sell Minds.com to a corporation, people who might face legal, financial and political pressures, and people who would eventually be hiring others.

When asked, by Neoxian, writing for Steemit, whether Minds could truly be considered decentralised, Ottman gave the following answer:
‘Good questions. It’s decentralized in that ultimately, yes, nodes will be able to optionally federate (this is still in dev). It is censorship resistant in that we allow all legal content, and in the future will integrate torrent options.‘

This is actually not an empty promise. The Minds developers have already been working on a decentralisation component called ‘Nomad‘, which is based on the Beaker browser and the DAT protocol. I’ve experimented with these briefly this weekend, and they really do work. If a P2P system does go mainstream, it’s likely to be this.

John Keats and State Surveillance

Tags

Keats, Literature, surveillance

Dr. Richard Margraff Turley is due to give a talk on Literature and Mass Surveillance at the BCS Mid Wales event this week (Aberystwyth University is a four-hour drive for me, unfortunately). Titled ‘Who hath not seen thee … ?‘, this is an interesting discussion about the idea that state surveillance had some influence on the later works of John Keats.

The relevance of Keats’ writings to mass surveillance aren’t obvious. I think it’ll become clear to some readers that Keats, if he was indeed commenting on surveillance, was trying to describe a situation that was very different to that of today, and Dr. Turley hasn’t differentiated between mass surveillance of today and targeted surveillance of the past. The two are very different:

‘We might assume mass surveillance is a modern phenomenon, but “surveillance” is a Romantic word, first introduced to English readers in 1799. It acquired a chilling sub-entry in 1816 in Charles James’s Military Dictionary: the condition of “existing under the eye of the police”.‘

This definition, though very concise, is remarkably broad. What precisely does it mean to be ‘under the eye of the police’? It could refer to a state in which reasonable suspicions are investigated by detectives (targeted surveillance). It could refer to a state in which those ‘with nothing to hide’ are watched by a myriad of deputised officials and machinery (mass surveillance). Mass surveillance is something that violates our reasonable expectations of privacy in the most insidious ways.

It’s also important to remember that Keats penned his works in a politically volatile period, just a few decades after the revolutions in France, America and Haiti. Masses of people were awakening to the fact they could revolt and potentially overthrow governments in their struggle for universal rights, suffrage, a better quality of life, and even their very survival. Today the opposite is true: Most of us have everything to lose and little to gain by overthrowing The Establishment. Also, today mass surveillance wouldn’t be viable in a Western society without Silicon Valley corporations and and social media to provide the framework.

So, it wasn’t without reason that The Establishment would have employed spies to watch public events for indications of an imminent uprising, and resorted to heavy-handed tactics to prevent that happening. The Establishment felt it necessary to mobilise the police and the Army to protect the Bank of England, among other buildings, and charge radicals like Henry Hunt with treason. Could we really claim that as an example of mass surveillance, though? No, I think the issue here was that the use of state surveillance to monitor the political activities of citizens, instead of, say, people who already had political influence, was a new concept at the time.

What’s more telling than the content of the literature is the way Keats was guarded in his commentary of events, as if he suspected The Establishment knew about his more politically active acquaintances and were intercepting his letters. Today we refer to this as the ‘chilling effect’ – the reluctance to openly express dissenting opinions for fear of retribution. This is not an irrational fear when political discourse is divisive and uncivil.

Lamia

Of Keats’ poem, ‘Lamia, Dr. Turley writes: ‘That poem opens with a queasy scene in which Hermes transforms Lamia from serpent to woman. The price is information: Lamia agrees to give up the location of a nymph’s “secret bed” to the priapic god.‘

Again, it’s conjecture to say that Keats was making a veiled reference to The Establishment’s surveillance apparatus, but we could nevertheless read that section of the poem as an allegory for it. Keats seemed to recognise and allude to the fact that people are willing to betray secrets in return for something, for some kind of benefit, rather like we’re collectively prepared to trade personal information for our 15 minutes of fame on social media. The nymph’s ‘secret bed’ could be a metaphor for a place where dissidents conspire, but it could also be a warning that even intimate details about ourselves and others could be traded. And why does Hermes want the information? For his personal gain, obviously, not for selfless reasons.

And once that level of sharing becomes accepted behaviour, it can quickly become a habit of inadvertent disclosure, as Keats and Turley also noticed:
‘Keats is describing actual workers, real people whose slacking off he reports as unthinkingly as we might share our own peers’ political views or locations on social media. As casually as a Google car might capture a moonlighting worker up a ladder outside someone’s house.‘

The vim Text Encryption Feature

Tags

Blowfish, Blowfish2, configuration, encryption, PkZip, Text Editor, vim

Using vim, it’s possible to save files as ciphertext. Simply use the ‘:X‘ command with the file opened in vim, set the password and save the file. There’ll be the following warning displayed when setting the password in the default encryption mode:

Warning: Using a weak encryption method; see :help 'cm'

Changing the Encryption Mode

Looking at the help page (‘:help cryptmethod‘), we find that vim supports only three ciphers by default: PkZip, Blowfish and Blowfish2. PkZip is considered weak – it is a stream cipher that XORs the file with the password. Blowfish’s implementation in vim is dodgy, with multiple bytes repeated in the ciphertext. Blowfish2 is an improved and recommended option, but older installations of vim cannot decrypt a Blowfish2-encrypted file.

It’s possible to set the encryption type for an opened file with either of the following commands:

:setlocal cm=blowfish2
Or
:set cryptmethod=blowfish2

To set the encryption mode as a configuration option, add the following lines to /etc/vim/vimrc:

set cm=blowfish2
set viminfo=
set nobackup
set nowritebackup


This should set the default cipher to Blowfish2, and prevent any background writes of the plaintext to the hard disk.

A Solution for Synchronising and Sharing Encrypted Text

Tags

encrypted, encryption, ProtectedText, Safe Notes

The SafeNotes application caught my interest the other day, while casually browsing the Play Store. Initially I thought it was simply the usual password protected notes thing with a nice UI. The application is pretty straightforward to use. Just press on the key icon to set a password for a note, and the padlock icon to encrypt or decrypt it.

Then I noticed the ‘Add existing note‘ option. It turns out the application can be used to sync notes with a site called ‘ProtectedText.com‘, that stores the ciphertext.

ProtectedText.com

Visiting the URL, we get a rather basic-looking site, which I haven’t come across before. It’s a service for storing encrypted notes, which can be used with or without Safe Notes. The encryption is handled by a main.js file which encrypts and decrypts the content locally, so the plaintext should never be stored on the server.
I’ve briefly examined the JavaScript, and it appears pretty legit. A WireShark capture during a browser session with one of my sites here revealed nothing in plaintext beyond a DNS lookup for protectedtext.com.

Each text file is referred to as a ‘site’, and there is a reason for this. Entering a unique name in the text field will create a new ‘site’ at the URI, and each tab could be viewed as an individual page. If several people know the full address and the password, they could use the site as a ‘drop box’ or a private Web site.

To sync the Safe Notes application with a site, just press the ‘Add existing note‘ option, enter the note’s URL, and then enter the password for it.

After the application is synched, it’s simply a matter of uploading or downloading changes. I’m not sure if there’s a version control system for multiple users.