Data Wrangling Course

James Howison's Data Wrangling course from the Information School at the University of Texas at Austin.

This semester we’re going to be using the Atom programming editor. A good programming editor has many advantages, including:

There are many options out there. In previous semesters I’ve used TextWrangler but since that is not available for Windows I’ve decided to standardize on Atom.

To upload files we will use the remote-ftp package for Atom. This lets you edit and create files on the server.

Once you have it installed you have to create a folder on your local computer which will contain the information Atom needs to connect to Holden. In my examples below I’ve created a folder called 2017Demo. You can create a folder by choosing File -> Open and then using the normal method of creating a folder on your computer (on the Mac that’s the New Folder button). One caveat: make sure the folder you create is a local folder on your computer, not a cloud folder (i.e., don’t put this on iCloud or Dropbox etc.).

After you create the folder you’ll see a window that looks like this:

Then we need to create the file that will hold the connection information. To do that we’ll open the SFTP column, using Packages -> Remote-FTP -> Toggle:

This opens a column which will eventually show the remote files. So you have your local folder on the left, and your remote files in the middle, and we’ll have the file we’re actually editing show up on the right.

To create the config file we go back to Packages -> Remote-FTP -> Create SFTP Config File which will create file in our local folder and bring it up on the right of the screen.

See the file .ftgconfig which is now in our local folder?

You will need to copy the details below into this file (erasing what is there already) these config variables into your .ftpconfig file. The account is the one you created on James’ laptop. user is your holden username (not your EID), pass is your holden password, and remote is the path to your home directory. To find out the needed path log into holden using ssh and type pwd.

{
    "protocol": "sftp",
    "host": "holden.ischool.utexas.edu",
    "port": 22,     
    "user": "testuser", # your_holden_username.
    "pass": "north-forensic-panic", # your_holden_password
    "promptForPass": false,
    "remote": "/home/testuser" # Output_of_pwd_command
}

You then need to save this file. At that point we can select Connect from the middle SFTP tab. If everything goes to plan you’ll see this:

If that doesn’t come up check:

  1. That you saved the .ftpconfig file after editing it.
  2. That the file contains the details above.
  3. That you are not using your EID but the user/pass combination you created with James for holden.

Once connected we can create files or folders using a right-click on the remote folder:

Right-click and choose Add file and that will bring up a window that says “Enter the path for the new file”:

Here you can type a filename, I typed “testfile”. You should see a new file called “testfile” in the top-level of your home folder, showing up in the middle column and in a new tab, ready for editing.

You should also be able to see this file using the command ls in Terminal:

Go ahead and create some files and folders using right-click in the ftp column (middle column) and run ls to see them in Terminal. You can also right-click on folders you created to create files inside them. To see those files on the Terminal you have to use cd to change directory and then ls.

Moving files from computer to the server

To move files between your computer and the server you have a number of options.

  1. Use the Remote-FTP module in Atom. This works well for creating and then editing files. For simple files you can open the file in Atom, then copying the content and paste it into a file you just created. Atom Remote-FTP also has a “Sync local -> remote” function hidden under the right click menu. You have to get the file you want to move into the same folder as your .ftpconfig file (or one of the sub-folders) then choose “sync local –> remote”.

  2. Use a fully fledged SFTP client. A decent free client on the Mac is Fugu. Be sure to choose the “Universal-English” option. You may have to go to your security tab in System Preferences to allow install. Fugu is a bit old, so another option is Fetch, which provides a free version for students. Filezilla is another option which also works on Windows (although be aware that the installer asks you whether you want to install additional software, you can decline the additional software.) For both of these you need the login information above.

The iSchool has some tutorials:

  1. Use scp on the commandline. scp stands for secure copy or SFTP copy. There are tutorials available online, e.g. https://linuxacademy.com/blog/linux/ssh-and-scp-howto-tips-tricks/. You will use the same connection details as setup for Atom sftp.