Follow-up: a BASH script to split a MySQL dump by database

In this post, we’ve seen how to split a large MySQL dump by database.

I’ve been asked a script to automate the process. Here you are.

Note: On FreeBSD, replace AWK=awk by AWK=gawk and install lang/gawk port, so we can use GNU awk.

Split a large SQL dump by database

You created a MySQL backup of a large server installation with dozens of databases and wish to get the schema and data for one of them. You now have to deal with a file of hundreds of MB in a text editor. How convenient.

Split a dump into several files

You can quickly split this dump in several files (one per database) with awk or csplit. With GNU awk (gawk on FreeBSD), this is a oneliner:

Get database.sql files

To rename these files with actual database names, the following bash script could be useful. It assumes you don’t have the main dump in the same directory.

Chromebook: run a SSH server on Chrome OS

In this post, we’ll cover how to run a SSH server directly on Chrome OS (ie not into a Crouton chroot).

One of the first things I do on any machine (FreeBSD, Linux, Mac OS X or Windows) is to install, run and configure the SSH server. It’s always convenient to be able to scp from and to a computer, or to log in remotely. Even for workstations.

Chrome OS is a reasonable if minimal standard Linux installation offering access to iptables and sshd (and openvpn by the way), so it’s as easy to run sshd and to allow incoming traffic on port 22.

Setup

1. If it’s not already done, switch your chromebook in developer mode, so you can execute commands as root.

Do a backup of your data, as you’ll wipe your current Chrome OS partitions.

On most recent machines, restart in recovery mode (ESC + REFRESH + POWER), then when it boots, CTRL + D to enter the developer mode.

Hit enter to turn off OS verification. It will then restart. Now and everytime after, you’ll need to do a CTRL + D to boot.

It will then wipe your chromebook and reinstall a fresh Chrome OS version. The process takes 6 to 7 minutes.

Former machines require to use an hardware switch, generally located below the battery. Be gentle with this switch, it breaks easily.

2. Launch a console with the shorcut ctrl + alt + t, then write shell to open a full bash shell (if the shell command isn’t available, you aren’t in developer mode).

Become root with sudo su.

3. Setup SSH keys :

4. Run SSH:

5. Allow world to connect to port 22:

6. Add your public keys to ~chronos/.ssh/authorized_keys file. Authentication by password isn’t available.

7. You’re now able to log in from the world to your chromebook.

Sources

Andrew Sutherland, cr-48 chromium os ssh server, 14 January 2011.

CentOS wiki contributors, IPTables, CentOS wiki.

December 2014 links

Some links of stuff I appreciated this month. Links to French content are in a separate post. You can also take the time machine to November 2014.

AI

What if instead to understand how the brain works, we copy the neural connections as is? This is what the OpenWorm project tries to do with C. elegans. And, big surprise, that works and allows a bot to move.

Wikipedia

An infographics of the locality of Wikipedia participants shows without any surprise they are mainly from Europe and North America.

If you’re into dumps, the Wikipedia / MediaWiki XML dump grepper will help you to find a particular piece of data, like the text of one article.

Tools

Dev / search. The silver searcher, ag, offers a faster approach than ack to search your code.

Fun / autogenerator. Some years ago, cgMusic offered an implementation on how a computer program could create music. Add some image generation techniques and a word generators, and you can have a fake music generator offering full albums. Ælfgar has stumbled upon Liquified Death by Income Yield.

GIS. Turf is a new open source JavaScript GIS library. This post explains the capabilities and features, including its great offline support.

Electronics

What if an Arduino embeds a web server and allows programmation from the web browser? This is exactly what the Photon by Spark does.

Quartz

An infographics showing satellites orbiting Earth and a point of view of the Uber economy.

Literature

The GoT series offer some comprehensive scenes of torture. Did you ask yourself their interest or need for the plot? Marie Brennan offers a great opinion in « Welcome to the Desert of the Real ».

November 2014 links

Some links of stuff I appreciated this month. Links to French content are in a separate post. You can also take the time machine to October 2014.

November is the Philae landing on the Comet Churyumov-Gerasimenko month and the ESA photo release under CC-BY-SA (one of them here) month. Mainly DevOps links in this post, a Wikidata tool and an algorithm visualisation.

Churyumov-Gerasimenko 67P, 20 November 2014
ESA/Rosetta/NAVCAM, CC-BY-SA 3.0 IGO

Dev

Craft. Jeroen de Dauw has prepared interesting slides about clean functions. Your function should do one task, not be a class disguised in procedural code.

Raft. In a distributed environment, how do you achieve a similar state? Raft is an answer to this question, as a distributed consensus algorithm.  To understand how it works, The Secret Lives of Data offers a visual guide.

Wikidata

Wikidata no labels. Harmonia Amanda and Hsarrazin wanted to find items without labels in French, respectively about the Tolkien’s Legendarium or Russians persons to translate. This tool allows you to get some Wikidata items through a WDQ query or to encode them directly, and print a table with the part of these items without label in the specified language.

DevOps

Once upon a time there were a Linux theme park. As a Cobbler / SpaceWalk alternative, we start to see new software to appear: katello/foreman. It’s a part of Katello, the upstream of Satellite 6, and a replacement for SpaceWalk. You want to dive into the Linux theme park? Build images, deploy, manage resources? You’ll be served. Thank you to jnix for these software recommendation.

And now, near the sea. ShipYard allows you to manage Docker instances and containers.

But what is more interesting is the alpha release of OpenShift Origin, the third generation of  OpenShift, with a new system design. It relies on Docker and the following technologies:

  • Kubernetes, an active controller to orchestrate and ensure the desired state of the containers;
  • An etcd server (which uses the Raft algorithm described above);

With that concepts, you’re ready for the introduction hands-on tutorial available.

The puppetmaster becomes old. Ryan Lane, formerly in Wikimedia ops team,  blogged this summer about a Puppet alternative at his new job: Moving away from Puppet: SaltStack or Ansible? For Ryan, 10K+ lines of Puppet codes is now only 1K of SaltStack or Ansible code. The winner of their test to port the Puppet infrastructure into both is SaltStack. It’s a pity, I would have loved to merge yet another fictional universe into the Nasqueron project and add the Ursula K. Guin ansible in the mix.

Sysadmin

FreeBSD 10.1. The first new version of FreeBSD after the SSL bugs is out, and will immediately be deployed on Ysul and Sirius machines as test. Bhyve can use a pure ZFS filesystem and UDP-Lite protocol is finally here.

TCL and the SSL security issues: sslv3 alert handshake failure

Update 2016-01-15: With tcl-tls 1.6.7, it works out of the box without any need to configure cyphers.

If you have reconfigured your OpenSSL to take care of the current security issues, you’ve disabled SSLv3 since POODLE discovery.

Then, you could find unexpected behavior of TCL code. The package http isn’t the best to intercept and report errors, so it could be as non descriptive as software caused connection abort. If you’re luck you’ll get the actual cause of the error sslv3 alert handshake failure.

So, without any surprise, we disabled SSLv3, code still want to use SSLv3, and… that fails:

[sourcecode language=”plain” highlight=”9-10″]
/home/dereckson ] tclsh8.6
% package require http
2.8.8
% package require tls
1.6
% http::register https 443 ::tls::socket
443 ::tls::socket
% http::geturl https://fr.wikipedia.org/
SSL channel "sock801eacd10": error: sslv3 alert handshake failure
error reading "sock801eacd10": software caused connection abort
[/sourcecode]

The solution is to explicitly request to use TLS.

[sourcecode language=”plain” highlight=”6″]
% /home/dereckson ] tclsh8.6
% package require http
2.8.8
% package require tls
1.6
% tls::init -tls1 true -ssl2 false -ssl3 false
-tls1 true -ssl2 false -ssl3 false
% http::register https 443 ::tls::socket
443 ::tls::socket
% http::geturl https://fr.wikipedia.org/
::http::1
% http::cleanup ::http::1
%
[/sourcecode]

In your TCL application, register once for all the https as preconfigured TLS socket sounds a good idea:

[sourcecode language=”plain”]
#
# HTTP support
#

package require http
package require tls
::tls::init -ssl2 false -ssl3 false -tls1 true
::http::register https 443 ::tls::socket
[/sourcecode]

Thank you to rkeene from Freenode #tcl for his help to track this issue.

October 2014 links

Some links of stuff I appreciated this month. Links to French content are in a separate post.

In the servers world

SSL. October is the month we disabled SSLv3 protocol support from nginx following the POODLE attack. So this means we can look to this paper, nginx configuration and a tool to check SSL configuration. The provider Linode has published a comprehensive guide to mitigate the attack.

FreeBSD. FreeBSD 10.1-RELEASE will soon be available. The virtual terminal console driver vt is improved. Oh, and you can now boot bhybe on ZFS. Shell servers will have to deal with the fact login.conf settings will take precedence on .profile and other shell environment for variables like path, blocksize or umask.

Docker. To improve Docker workflow, nitrous.io has released tug, a set of scripts in Go to help common tasks.

Thus shall ye compile in JavaScript

Humble Bundle launches the Humble Mozilla Bundle, games compiled in ASM.js and so playable in the browser.

Meanwhile, in the functionnal language world, a paper shows you can compile OCaml in JS, an it’s sometimes quicker in the JS JIT than it its own JIT (but well… you can also compile OCaml in native, and OCaml JIT isn’t really well optimized).

So if you want to respect this commandment, just compile your C code with clang: emscripten will then happily compile your LLVM bytecode in ASM.js.

Gamergate / NotYourShield

A CNN journalist reads the gamergate as the end of the narration controlled by journalists.

When an Examiner journalist suggests #NotYourShield is 4chan white heterosexual users posing as women and PoC, his tweet is replied with a lot of photos from women and PoC. We so now have a picture of the diversity in video games (permanent link).

On a related theme, I Can Tolerate Anything Except The Outgroup is interesting to read and heavily commented.

Finally, a call for help:

Curiosities

Some scientists push to a new definition of planet, to take in account exoplanets. In such a definition, Pluto would be again a planet. Harvard organized a debate, this position wins.

At Databricks, they carved this pumpkin for halloween:

How to change Phabricator logo

Before summer 2016, Phabricator didn’t provide a quick way to change the logo. In the software past, the possibility was also offered, but between July 2012 and 2016, the logo was a resource embedded into new sprites, with other header graphics.

You can track the issue on Phabricator’s Phabricator.

Meanwhile a solution is provided by the product, here the procedure to update the UI and change the logo.

Continue reading “How to change Phabricator logo”

Install Final Term on Debian

Final Term is a new terminal application, currently under development,  written in Vala and built on top of GTK+ 3, Clutter and Mx.

Screenshot of Final Term on Debian Jessie.
Screenshot of Final Term on Debian Jessie. The bottom bar is from tmux.

If you can install it easily under Ubuntu through a PPA package, it’s not the case for every OS. It’s packaged for Fedora and downstream. It’s also available in Linux distribution with alternative packages repositories for second-class citizen or denizens packages in Arch AUR, and for Gentoo as a Project SunRise ebuild. These links have been prepared as writing time, and they don’t really look as permanent URLs.

Let’s see how to install it on Debian.

Continue reading “Install Final Term on Debian”

Grip end of life. Migration to Dwellers and Ysul.

We’re migrating this summer our semi-public web and shell server Grip, so we can provide  more space disk, more RAM and better CPU.

Grip EOL date is 2014-07-31. Alternative servers are ready now.

The new offer

We both offer a traditional FreeBSD shell and web server and a CentOS 7 application server with OpenShift and Docker capabilities. The first is named Ysul and the second Dwellers.

Services are provided free of charge and are primarily targeted to both relocate the former Grip users and host the Nasqueron projects. These new servers are a part of the Nasqueron infrastructure.

Track issues and request assistance

A Grip migration project has been created on our Phabricator instance, with a kanban board to get an overview of what is still to be done.

You can get assistance on IRC Freenode #Wolfplex. You can also create a migration task.

Where to migrate?

As a Grip user, you can:

  • migrate any non-IRC shell and general web services to Ysul
  • migrate any services to a Docker container or an OpenShift gear on Dwellers

IRC services hosting is currently discontinued, as we don’t have an IRC friendly ISP on the matter. We’ll work in the following months to find solutions on this issue.

Typically, any Python, Node or Ruby webserver will be hosted on Dwellers, static and PHP sites on Ysul.

If you need or could benefit of root control for your application, this is also a use case for a Docker container.

For example, this blog is on Ysul, but the Phabricator instance with the migration board is on Dwellers.


2014-08-03 update: wiki migration page has been replaced by the Phabricator instance’es migration board link.