CSE330 Wiki - User contributions [en]

Module 2

2024-09-03T06:14:13Z

Zach: /* Reading */

__NOTOC__
Module 2 introduces you to Linux, a command-line environment, the Apache web server, and PHP. You will create and configure your own cloud instance, install Apache and related software, and then form groups to make a simple file sharing site.

<span style="font-size:1.3em; line-height:1.2em; color:blue;">'''DO NOT WAIT UNTIL THE LAST MINUTE TO START ON THIS (or any) MODULE!''' The most common reason students perform poorly in this class is procrastination. You have been warned.</span>

== Using the Wiki ==
{{RequiredInstructions|content=
Text enclosed by <syntaxhighlight lang="bash" inline><</syntaxhighlight> and <syntaxhighlight lang="bash" inline>></syntaxhighlight> should be replaced by content unique to you.

'''Example'''

<source lang="bash">
$ sudo useradd -r -m -c "<My Full Name>" <usernameHere>
</source>
becomes
<source lang="bash">
$ sudo useradd -r -m -c "Zach Cohn" zcohn
</source>
}}

== Reading ==

The following articles on the online class wiki textbook contain information that will help you complete the assignments.

''To jump right into setting up your EC2 Instance, Begin with [[SSH]]''

* [[Linux]]
* [[Bash]]
* [[Amazon Web Services]]
* [[SSH]]
* [[Apache]]
* [[PHP]]
* [[Workflow]]
* [[VS Code]]
* [[Web Application Security, Part 1]]
* [[FAQ - Mod 2]]

== Individual Assignments ==

{{RequiredInstructions|content=

<span style="color:red">There are a lot of wiki pages that you need to read in order to complete the individual portion of Module 2. The wiki pages contain an assortment of required steps, optional steps, and information. To make things easier, we have highlighted the ''required steps'' with a red bar, like you see on the left of this message. Do not skip anything marked with a red bar, or else you will not have a perfectly configured instance.</span>

}}

=== Learn About Linux ===

Linux is an open-source operating system based on UNIX. Linux is highly versatile and is used in a wide range of applications.

If you are not a Linux guru, read the Linux guide to get started: [[Linux]]

=== Windows Users Only: Enable Windows Subsystem for Linux (WSL) or Install Cygwin ===

You will need WSL or Cygwin in order to log into and configure your EC2 instance. Cygwin is already installed on the CEC machines. If you're using Windows 10, you can use WSL, which is an officially-supported way of accessing a Linux terminal on Windows. If not, you'll have to use Cygwin. Instructions for installing either on your personal machine are located in [[Workflow#Cygwin|the Workflow guide]].

Mac OS X users should located the "Terminal" application, which is installed by default; you will need this in the coming steps.

=== Create an AWS EC2 Instance ===

Once you understand [[Linux]], you need to set up your Amazon EC2 Instance. Use the AWS article to guide you through the process: [[Amazon Web Services]]

=== Configure SSH ===

Follow the instructions in [[SSH#SSH Configuration|the SSH guide]] to do the following:

# Create your own SSH key pair
# Log in as the default user to your EC2 instance
# Create a new user
# Give the new user your SSH public key

From now on, when you need to log into your EC2 instance as an administrator, just pop up a terminal and run:

$ ssh <your-instance-username>@<ec2-xx-xx-xx-xx.compute-1.amazonaws.com>

To save from typing this command in every time you want to access your server, you can set an '''alias''' in your terminal.

In the '''Mac''' terminal, vim ~/.zshrc and input the following command:
$ alias aws='ssh <your-instance-username>@<ec2-xx-xx-xx-xx.compute-1.amazonaws.com>'
Now, whenever you type aws in your terminal, it will type your ssh command.

In '''Cygwin''', vim ~/.bash_profile and input the following:
export aws="ssh <your-instance-username>@<ec2-xx-xx-xx-xx.compute-1.amazonaws.com>"
And you run this command by typing '''$aws'''
Also note that you may have to restart your terminal for this to take effect, or put the command in your ~/.bashrc file

=== Install Essential Packages ===

Recall that software installation in Linux instances is usually handled through a package manager, like '''yum'''. For more information, refer to [[Linux#Repository-Based Package Managers|the Linux guide]].

To make sure everything is working properly, '''install your first package(s) according to the ''Essential Packages'' subsection in the Linux guide.'''

=== Set the Timezone ===

'''Set your server to use US Central time (America/Chicago).''' Instructions are in the Linux guide: [[Linux#Synchronizing Date and Time]]

=== Set Up the Apache Web Server ===

You need to '''install the Apache web server on your EC2 instance'''. Refer to [[Apache|the Apache guide]].

In order for your web server to be accessible, you need to '''open up Port 80 on your EC2 instance'''. Instructions are in the AWS guide: [[Amazon Web Services#Enabling Web Access to your EC2 Instance]]

'''Enable the UserDir module in Apache.''' Instructions are in the Apache guide: [[Apache#The UserDir Module]]

=== Install PHP ===

You need to '''install PHP on your EC2 server'''. Instructions are at the top of the PHP guide: [[PHP]]

You need to '''configure PHP to show errors'''. Instructions are immediately below installation in the PHP guide.

=== Learn about the PHP Language ===

Before you can continue, you will need to make yourself acquainted with PHP language components. Read the PHP guide for an overview: [[PHP#PHP Language Components]]

=== Set Up your Workspace ===

You are now ready to set up the workspace that you will be using for the rest of the semester. Follow the instructions in [[Workflow]] to:

# Make an account on GitHub
# Setup a repo using GitHub Classroom
# Clone your CSE 330 repo from GitHub
# Install a text editor such as '''VSCode''' to write your assignments with
# Either install an FTP client, or use SFTP through command line, to transfer files from your local machine to your instance

=== Calculator ===

'''Make a calculator using PHP and an HTML form.''' The form should have two number inputs. The form should submit a GET request either back to the same page or to a different results page. The calculator should support addition, subtraction, multiplication, and division by means of a radio button group on the page. (Consider edge cases!)

=== Git Tutorial ===

Complete all of the levels of the Main and Remote sections of [https://learngitbranching.js.org/ this Git tutorial]. After you finish the tutorial, take two screenshots showing the completion of all of the levels of the Main and Remote sections. Place these two screenshots in your Module 2 Individual repo.

== Group Project ==

You will work in pairs (that means you and one other person) on this project.

<span style="font-size:1.3em; line-height:1.2em; color:blue;">In this module, and all future modules, the group portion builds off of material you learn in the individual portion. You should therefore complete the individual portions prior to working on the group portions. (Trust us: it will make your life easier!)</span>

'''Important Reminder:''' frequently commit your work to your repository as a backup!

=== Simple File Sharing Site ===

You will be making a simple file sharing site that supports uploading, viewing, and deleting files associated with various users. Details:

* You should have a file named '''users.txt''' stored in a secure place on your filesystem. It should have at least three usernames, with one username per line.
*: Passwords add an extra layer of complexity (encryption) that we will cover in Module 3. You will not earn credit for implementing passwords as your creative portion for this module.
* Users of the file sharing site should be able to enter their username and then log in.
*: You may implement logins using session variables, or you may implement them using a GET parameter that is passed between pages. You will need to always use sessions starting in Module 3.
* Users should see a list of all files associated with their username after the log in.
* Users should be able to view, upload, and delete files associated with their username.
* The URI should NOT reveal the internal file structure of your web site.
** Example of what not to do: http://www.example.com/download.php?file=/home/paul/file.txt
** Another bad example: http://www.example.com/~user/file.txt
** Good example: http://www.example.com/view.php?user=music&name=file.txt
*: In general, don't reveal any internal information about the site.
*: '''Food for thought:''' Given that Apache processes (with the exception of the main process) do not run as root, but rather as the user apache, how can you allow both your user account and the php scripts to read and write the
*:necessary files?
*:: ''Hint:'' You shouldn't be storing the user-uploaded files in public_html (that is only the place for your own web-page files),
*::''Hint number 2:'' you will need to use chown to change permissions to let user apache have permissions on the user-files (that '''are not''' in the public_html folder). If you don't know what this does or how to do this, look at the [[Apache]] page
* You should customize your file sharing site by implementing an additional feature or two as part of the '''creative portion'''.

You will probably find the PHP guide on this wiki to be helpful: [[PHP#Other PHP Tips]]

=== Web Security and Validation ===

Your project needs to demonstrate that thought was put into web security and best practice. For more information, see this week's Web Application Security guide: [[Web Application Security, Part 1]]

In particular:

* '''Your application needs to follow the conventions of FIEO (Filter Input and Escape Output)'''. Things you should filter include file names, usernames, and so on.
* '''Your project must pass the W3C Validator with no errors or warnings'''. For more information, see the HTML and CSS guide: [[HTML and CSS#Validation]]

== Grading ==

We will be grading the following aspects of your work. There are 120 points total.

'''Assignments (including code) must be committed to GitHub by the end of class on the due date. Commit early and often. Failing to commit by the end of class on the due date will result in a 0.

_____________

# '''Cloud Instance and Web Server (30 Points):'''
#* You can SSH into your instance using your own custom username and SSH key. Put a screenshot in your repository of your terminal SSH-ing into your instance (15 points)
#* Your custom username is able to perform SUDO commands. Put a screenshot in your repository of you successfully executing a command with sudo (5 points)
#* The time zone is correct on your instance. Put a screenshot in your repository of the results of the `date` command (5 points)
#* The UserDir Apache module is working. Place a file in /home/<username>/public_html and link to the file in your README.md (5 points)
# '''PHP Calculator (15 Points):'''
#* The calculator is able to perform all four functions with floating point numbers (2 points each, 8 points in total)
#* Each function is given as a radio button choice (4 points)
#* The calculator page passes the W3C HTML validator (3 points)
#* '''''Make sure you have a README.md file in your individual repo with the link to your calculator on your instance.''''' See the Creative Portion section for more information about .md files.
# '''Git Tutorial (20 Points):'''
#* Two screenshots showing the completion of all of the levels of the Main and Remote sections placed in your individual assignment repository.
# '''File Sharing Site (40 Points):'''
#* '''''File Management (25 Points):'''''
#** Users should not be able to see any files until they enter a username and log in (4 points)
#**: ''Remember that users.txt should be stored in a secure location on your filesystem. That is, '''you should not be able to type any URL into your browser and see the raw users.txt file!'''''
#** Users can see a list of all files they have uploaded (4 points)
#** Users can open files they have previously uploaded (5 points)
#**: ''Note: Users should be able to open not only plain text files but also other file formats: images, spreadsheets, etc.''
#** Users can upload files (4 points)
#**: ''Note: Like users.txt, uploaded files should be stored in a secure location on your filesystem. That is, '''do not keep your uploads directory underneath a directory served by Apache!'''''
#** Users can delete files. If a file is "deleted", it should actually be removed from the filesystem (4 points)
#** The directory structure is hidden. Users should not be able to access or view files by manipulating a URL. (2 points)
#** Users can log out (2 points)
#**: ''Note: If using session variables, you must ''actually'' log out the user by destroying their session; i.e., don't just redirect them to the login screen.''
#* '''''Best Practices (10 Points):'''''
#** Code is well formatted and easy to read, with proper commenting (4 points)
#** The site follows the FIEO philosophy (3 points)
#** All pages pass the W3C validator (3 points)
#* '''''Usability (5 Points):'''''
#** Site is intuitive to use and navigate (4 points)
#** Site is visually appealing (1 point)
# '''Creative Portion (15 Points)''' (see below)
#* '''''Make sure you have a README.md file in your group repo with the following:'''''
#** An .md file is a plain text file called a markdown file [https://guides.github.com/features/mastering-markdown/ ] .
#***The contents of your README.md file are shown on your repo, making it easier for us to grade
#*** You may style the page with bolding, hyperlings, images, block quotes and lists
#*** [http://dillinger.io/ This online markdown editor] makes it easy to learn this format
#** The link to your file sharing site. '''''We need this to grade your work.'''''
#** A brief description of what you did for your creative portion
#** Any additional login details needed for the TA

=== Creative Portion ===

This module and all future modules will require that you invest some time into creating additional features for your group project. Plan to invest at the very least 60 minutes of your time into the creative portion. In the above rubric, viewing files is worth five and uploading is worth four. The creative portion is worth fifteen points; it should be a very sizable part of your project.

* The creative portion is an opportunity for you to learn the material of your own interest.
* You will not earn credit for a creative-portion feature that simply rehashes something you've already done in a previous module.

If you need ideas for a creative portion, or if you want to know whether or not your creative portion idea is "hard enough", ask a TA.

[[Category:Module 2]]
[[Category:Modules]]

Node.JS

2024-03-22T23:01:41Z

Zach: /* Installing Packages */

AJAX is a useful technology for web applications that need to pull periodic updates from your server. But what do you do when you have thousands of clients all requesting data that changes real-time? A PHP+AJAX approach could choke without a lot of effort on your part. It is situations like these that Node.js comes to the rescue. Node.js is a web framework, made in JavaScript, that's become extremely popular over the last few years.

In general, JavaScript frameworks are very trendy right now; it can be hard to keep up with the new cool thing. In fact, there's a joke website https://dayssincelastjavascriptframework.com/, that "tracks" how long it's been since a new framework has come out.

== Non-Blocking I/O ==

You dabbled in the concept of non-blocking I/O when you implemented AJAX in the previous JavaScript module. Node.js uses this execution model at its core.

The core idea is the use of '''''anonymous functions''''' as ''callbacks'' for processing the results of intensive operations. Consider the pseudo-code:

<source lang="javascript">
// Blocking I/O Pseudo-Code

var data = database.query("select * from news");
print data[0].title;
</source>

The above pseudo-code queries a database, saves the result of the query in a variable named ''data'', and then prints out the first title in ''data''. The problem arises when multiple clients all want to access the server at the same time. ''In a blocking I/O model, a client needs to wait until all other clients are finished before it can start its process.'' With intensive operations and thousands of clients, this can easily crash your server!

The solution is to use ''non-blocking I/O''. Consider this revised pseudo-code:

<source lang="javascript">
// Non-Blocking I/O Pseudo-Code

database.query("select * from news", function(data){
print data[0].title;
});
</source>

The revised pseudo-code achieves the same end result as the first version. However, ''it allows other clients to perform tasks while this process is waiting for the database to be queried!'' The code occurs in two parts: first, we start the query. Second, ''when the query is complete'', our '''anonymous callback function''' is executed. In Node.js, callback functions are usually passed one or more parameters.

Better yet, with non-blocking I/O, you have no concurrency issues, because everything runs in just one thread. Nifty!

== Installing Node.js ==

Run the following commands to install Node and NPM:

<source lang="bash">
$ curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.3/install.sh | bash
-
$ source ~/.bashrc
-
$ nvm install 16
</source>



== Your First Node.js Application: Hello World ==

Let's try running the example application given on the front page of nodejs.org:

<source lang="javascript">
const http = require('http');
http.createServer(function (req, res) {
res.writeHead(200, {
'Content-Type': 'text/plain'
});
res.end('Hello World\n');
}).listen(3456);
console.log('Server running at http://localhost:3456/');
</source>

Paste the example into a file named `hello.js`, and then run it by calling `node hello.js`. You should now be able to see your Hello World by visiting your server at port 3456! (Press ^C to stop the application.) '''Note:''' If you're running the code from your EC2 instance, you will need to open port 3456.

Let's walk through each line in the above example.

* <code>var http = require('http');</code> enables us to use Node's built-in HTTP server. We've been spoiled that languages like PHP automatically load all of the libraries that we need. However, '''''most functionality in Node.js is NOT loaded by default.''''' When you want to use any non-fundamental feature of Node, you need to ''require'' it into a variable, as this example does with the HTTP package. You should be familiar with this concept when you needed to use ArrayList and other specialty packages in Java in CSE 132.
* <code>http.createServer(function (req, res) { ... }).listen(3456);</code> uses the HTTP server API to listen for HTTP connections on port 3456. The anonymous callback function will be called whenever a connection is made.
* <code>res.writeHead(200, { 'Content-Type': 'text/plain' });</code> specifies the Content-Type header to be "text/plain".
* <code>res.end('Hello World\n');</code> writes "Hello World" to the response.

Note the use of non-blocking I/O, even in this one little example. ''Whenever a connection is made, the callback function will be called and passed parameters associated with that connection.''

== A Static File Server Using Node.js ==

Node.js does not automatically serve up static files in the way that an Apache/PHP does. We need to either use an extension that handles this for us, like Express, or we need to write our own code to make a simple static file server. The latter option is really not all that difficult, and it gives good insight into how a static file server works.

=== Installing Packages ===

In order for the example code below to work, you need to install the ''mime'' package. To do this, '''cd to the working directory where you will save your Node.js JavaScript file''' and run:

<source lang="bash">$ npm install mime@3.0.0 --save</source>

''npm'' is the package manager for Node.js. Unlike all of the package managers we've used up to this point, ''npm'' installs dependencies locally, ''not'' globally. This means that you can have two different projects using different versions of the same package. ''npm'' will create a ''node_modules'' directory in the CWD and will install the packages into it.

'''Note:''' When using npm to install local dependencies, '''''do not use sudo'''''. It will screw up your file permissions.

=== Static Fileserver Code ===

The following code will make an HTTP server on port 3456 that serves up all static files in <STATIC DIRECTORY NAME> relative to app.js.

'''Make an attempt to understand this code.''' It's important that you are comfortable with the non-blocking nature, as well as how we handle file I/O.

<source lang="javascript">
// Require the functionality we need to use:
var http = require('http'),
url = require('url'),
mime = require('mime'),
path = require('path'),
fs = require('fs');

// Make a simple fileserver for all of our static content.
// Everything underneath <STATIC DIRECTORY NAME> will be served.
var app = http.createServer(function(req, resp){
var filename = path.join(__dirname, "<STATIC DIRECTORY NAME>", url.parse(req.url).pathname);
(fs.exists || path.exists)(filename, function(exists){
if (exists) {
fs.readFile(filename, function(err, data){
if (err) {
// File exists but is not readable (permissions issue?)
resp.writeHead(500, {
"Content-Type": "text/plain"
});
resp.write("Internal server error: could not read file");
resp.end();
return;
}

// File exists and is readable
var mimetype = mime.getType(filename);
resp.writeHead(200, {
"Content-Type": mimetype
});
resp.write(data);
resp.end();
return;
});
}else{
// File does not exist
resp.writeHead(404, {
"Content-Type": "text/plain"
});
resp.write("Requested file not found: "+filename);
resp.end();
return;
}
});
});
app.listen(3456);
</source>

=== Port 3456 ===

We are running our server on port 3456 as opposed to the default web server port, which of course is 80. We therefore need to open our Node.js application like this:

http://ec2-xx-xx-xx-xx.compute-1.amazonaws.com:3456/path.extension

However, this won't work on the first time. ''Why?'' (Try to figure this out on your own before reading the next paragraph.)

We need to open port 3456 on our Amazon EC2 instance. Remember when you opened port 80 way back in Module 2? We need to do the same sort of procedure again now.

# Log into your EC2 management console
# Go to "Security Groups"
# Select your security group from Module 2
# Under "Inbound", create a custom TCP rule for port 3456, and apply your changes.

== Node.js Security ==

Just because we're writing in JavaScript doesn't mean that we can forget about the security practices you've been learning in the last three months of CSE 330. XSS is still a problem, as is the use of ''eval'', CSRF attacks, and so on.

The two links below have good information about security vulnerabilities specific to Node.js that you should think about when crafting your application.

# Book from PragProg ($24, 2015): https://pragprog.com/book/kdnodesec/secure-your-node-js-web-application
# Slideshow from a Switzerland Conference on Application Security (2012): http://www.slideshare.net/ASF-WS/asfws-2012-nodejs-security-old-vulnerabilities-in-new-dresses-par-sven-vetsch

== Node.js Conventions ==

There ought to be a standard way to save metadata about your app, right? Well, ''npm'' introduces a straightforward way to do this in the form of a ''package.json'' file.

To make your ''npm'' file with a step-by-step prompt, run in your project root directory:

<source lang="bash">
$ npm init
</source>

One nifty feature of ''package.json'' is the way it automatically handles dependencies. For example, you can put the following lines in your ''package.json'':

<source lang="javascript">
"dependencies": {
"mime": "~2.4.6",
"socket.io": "~1.0.0"
},
</source>

Then, back on the command line, when you run

<source lang="bash">
$ npm install
</source>

it automatically installs the dependencies for you! No more headaches of trying to figure out which modules you need to install; all that information is saved for you in your code base.

'''Tip:''' Once you set up your ''package.json'' file, you can automatically add a dependency to it by using the ''--save'' argument on ''npm install'': <code>npm install --save socket.io</code>

[[Category:Module 7]]

Linux

2024-01-23T04:07:12Z

Zach: /* Adding a User to the Sudoers List */

[[wikipedia:Linux|Linux]] is an [[wikipedia:Free and open-source software|open-source]] operating system based on [[wikipedia:UNIX|UNIX]]. Linux is highly versatile and is used in a wide range of applications. [[wikipedia:Desktop Linux|Desktop Linux]] is Linux with a GUI (like Microsoft Windows or Mac OS X); Desktop Linux is popular in niche markets, and it is used widely in developing countries.

Linux is the most widely used operating system for web servers. In CSE330, we will be interacting with Linux from the command line. This article covers the tools you need to make the best use of Linux.

== Linux Distributions ==

The open-source community is responsible for the development of many different [[wikipedia:Linux distributions|distributions]] of Linux. Distributions, or ''distros'', are different "flavors" of the Linux operating system with different objectives.

There are hundreds of distributions of Linux. Three of the main branches are [[wikipedia:Debian|Debian]], [[wikipedia:SuSE|SuSE]] (based on [[wikipedia:Slackware|Slackware]]), and [[wikipedia:Red Hat Enterprise Linux|Red Hat Enterprise Linux (RHEL)]].

For CSE 330, you'll be using a version of Red Hat Enterprise Linux.

=== Red Hat Enterprise Linux (RHEL) ===

Red Hat Enterprise Linux, or '''RHEL''', was first introduced in 1994. RHEL is known for being a good choice for enterprises that wish to use Linux as their primary OS. RHEL has an abundance of administration tools.

The Linux Lab in Lopata Hall uses '''[[wikipedia:Fedora Linux|Fedora Linux]]''', a desktop distribution based on RHEL. '''[[wikipedia:CentOS|CentOS]]''' is a popular RHEL derivative that is widely used in web servers. The '''[[wikipedia:Amazon Machine Image|Amazon EC2 Linux AMI]]''' is derived from CentOS, making it a distribution in the RHEL family.

=== Special Note: Linux Kernel and Modules ===
{{XKCD
|id=619
|name=supported_features
}}

What separates Linux from other Unix variants is its kernel. The kernel is the most important component of the operating system and is responsible for scheduling processes, providing access to the hardware devices, allocating memory to the programs, and so on.

The Linux kernel uses both monolithic and modular approaches. A monolithic kernel is a single program that contains all the code so any addition to kernel (such as code to access a driver) requires recompiling the code. A monolithic kernel is usually a little faster and could have a smaller size since only the absolutely necessary code is there. The modular kernel, on the other hand, enables dynamic loading and unloading of kernel code, called ''modules''. Typical modules include device drivers. Thanks to this modular approach, Linux seldom requires a reboot after installing a new device.

== Bash ==

In this course, you will be interacting with Linux primarily on the command line. The most widespread ''command line language'' is '''Bash'''. This is described in more detail in [[Bash|the Bash guide]].

== Files and Permissions ==

At the core of a Unix-based operating system is a directory structure with files and permissions.

=== Filesystem Hierarchy ===

The root directory of Linux contains a dozen or so subdirectories, each with a specific purpose:

* '''/bin''' contains [[Bash#Running Programs|binaries]] used by all users
* '''/sbin''' contains system binaries typically used only by the system administrator
* '''/lib''' contains libraries for the binaries found in '''/bin''' and '''/sbin'''
* '''/etc''' contains configuration files
** '''/etc/yum.conf''' Configuration file for yum
** '''/etc/yum/yum.repos.d''' Directory containing .repo files for online repositories
** '''/etc/crontab''' System-wide crontab file
** '''/etc/fstab''' Information about default partitions to be mounted
** '''/etc/group''' List of groups in the system
** '''/etc/hosts''' List of IP addresses with their names
** '''/etc/inittab''' What to do at each run-level
** '''/etc/inetd.conf''' Configuration file for some internet services (replaced by xinetd.* in most systems)
** '''/etc/modules.conf''' Module information for the boot
** '''/etc/motd''' Message to be seen at the login prompt
** '''/etc/passwd''' User information
** '''/etc/profile''' System level initial file for sh and its derivatives
** '''/etc/shadow''' User passwords
* '''/dev''' contains device files
* '''/proc''' contains information on currently running processes
* '''/var''' contains files whose contents is expected to change
** '''/var/log''' contains system log files
*** '''/var/log/messages''' System/Kernel messages
*** '''/var/log/syslog''' System log (mostly for Daemons)
*** '''/var/log/wtmp'''' User access log (binary)
*** '''/var/log/dmesg''' Boot-up messages
*** '''/var/log/auth.log''' Authorization logs
** '''/var/lib''' contains packages and database files
** '''/var/spool''' contains print queues
* '''/tmp''' contains temporary files that are deleted at system reboot
* '''/usr''' contains user programs
** '''/usr/bin''' contains binaries for user programs
** '''/usr/sbin''' contains binaries for system administrators
** '''/usr/lib''' contains libraries for '''/usr/bin''' and '''/usr/sbin'''
** '''/usr/local''' contains programs that you install from source
* '''/home''' contains users' home directories
* '''/root''' is root's home directory
* '''/boot''' contains boot loader files (do not touch unless you know what you are doing!)
* '''/opt''' contains optional add-on applications
* '''/mnt''' is where system administrators can mount filesystems
* '''/media''' contains links to removable media devices (for example, CDs)
* '''/srv''' contains site-specific data which are served by the system

For more information, see the Wikipedia article on the [[wikipedia:Filesystem Hierarchy Standard|Filesystem Hierarchy Standard]].

=== File Permissions ===

Every file in Linux has permissions that define which users can Read, Write, and Execute it. Every file has an owner and a group. The permissions for a file are set on three levels: User (owner), Group, and Other.

==== Symbolic Notation ====

When you view the permissions of a file in Linux, they will most often be displayed in ''symbolic notation''. Symbolic notation consists of 10 characters: the first defines the file type, and then there are three characters each for User, Group, and Other permissions.

* <code>-r--r--r--</code> is a normal file that is readable by all users but writable or executable by no one.
* <code>-rwxr-xr-x</code> is a normal file that is readable and executable by everyone but only writable by User (the file's owner). This is the most common permission set.

==== Viewing File Permissions ====

To view the permissions of all files in a certain directory, run the binary <code>ls -l</code> in Bash:

<source lang="bash">
$ ls -l # displays a list of all files in a directory with their permissions in symbolic notation
total 16
lrwxr-xr-x 1 sffc wheel 6 Aug 9 09:13 link -> myfile.txt
-rwxr--r-- 1 sffc wheel 12 Aug 9 09:13 myfile.txt
$ ls -l myfile.txt # displays the permissions of only myfile.txt
-rwxr--r-- 1 sffc wheel 12 Aug 9 09:13 myfile.txt
$
</source>

==== Setting File Permissions ====

Linux comes with several useful binaries for setting file permissions.

* '''chmod''' is used for setting permissions
* '''chown''' is used for setting a file's owner
* '''chgrp''' is used for setting a file's group

Some examples are shown below.

<source lang="bash">
$ chmod a+x myfile.txt # turns on the Execute option for all users
$ chmod o-w myfile.txt # turns off the Write option for Other users
$ chmod u+wx-r myfile.txt # turns on the Write and Execute options for User (the file's owner) and also turns off the Read option for User
$ chown todd myfile.txt # sets the owner of myfile.txt to the user todd. Note: First comes the user, then comes the filename: not the other way around!
$ chgrp staff myfile.txt # sets the group of myfile.txt to usergroup staff
$
</source>

For more information, see https://www.linux.com/training-tutorials/how-manage-file-and-folder-permissions-linux/

=== The . and .. Directories ===

The '''.''' directory is a reference to the current directory. The '''..''' directory brings you one level up in the filesystem.

=== Symbolic Links ===

A [[wikipedia:Symbolic link|symbolic link]], or ''symlink'', is basically a link from one spot in the filesystem to another. You can think of them like ''aliases'' in Mac OS X. To create a a symlink, use the ''ln -s'' command:

<source lang="bash">
$ ln -s /path/to/file.txt /path/to/link # creates a symlink to file.txt at /path/to/link

# Example:
$ ln -s /home/todd/instructions.doc /var/www/public_html/classes/instructions.doc # creates a symlink in the web server to instructions.doc
$ vi /var/www/public_html/classes/instructions.doc # changes to the symbolic link will be reflected in the original file
$
</source>

== User Management ==

=== Adding a New User ===

To create a new user, use the '''useradd''' command. Then, set a password for that user using the '''passwd''' command:

<source lang="bash">
$ sudo useradd -r -m -c "<My Full Name>" <myUserName>
$ sudo passwd <myUserName>
Enter new UNIX password:
Retype new UNIX password:
$
</source>

'''Note:''' If Linux doesn't like your command, first check your syntax and then try it with single quotes around your full name and password instead of double quotes. If this works it is likely that you used some special characters in your password such as '!', which serves a special purpose in the shell. This is because Linux takes parameters enclosed in double quotes as one string, but still looks for special characters inside of the string and acts in accordance with their normal meaning. On the other hand, single quotes tell the shell to ignore any special characters within.

=== Adding a User to the Sudoers List ===

For security reasons, you should never SSH into your server as the root user. Instead, you should use a normal user to whom you give '''sudo''' privileges. (For more detail on sudo, see [[Linux#SUDO|the Linux guide]].)

To give a user sudo privileges, use the command '''visudo''', which opens up the SUDO configuration file in the system's default text editor. (Never edit the file ''/etc/sudoers'' directly!) SUDO users are specified using lines similar to

alice ALL=(ALL) ALL

Add that line immediately below the line defining '''root''':

root ALL=(ALL) ALL

When you are finished, save and close the file. For more information on editing files on the command line, refer to [[#File Editors|the section about File Editors in this guide]].


In this case, '''alice''' will be able to run any command with SUDO privileges on the computer.

'''Note:''' The visudo utility is especially useful because it not only locks the sudoers file against simultaneous edits, but it also checks the file for syntax errors when you quit. Therefore, it is always recommended that you use visudo to change the sudoers list rather than attempting to modify it directly.

== Networking ==

In Linux, you can see your network information by typing '''ifconfig'''. This command shows the status information of each network interface, including the IP address you will need to remotely connect to your instance. The interface ''lo'' is the special loopback interface with IP address 127.0.0.1. This refers to your local machine and any connection from your machine to your machine goes through this pseudo-interface. Typical network interfaces include ''eth0'', ''eth1'',..., ''wlan0'', etc. Ethernet cards are represented with ''ethX''. In the past, most wireless cards showed up as ''wlanX'', but it is also common now for them to be represented with ''ethX'' names. ''ifconfig'' also gives information such as hardware address (MAC), netmask, and broadcast addresses.

You can start or stop networking by calling ''/etc/init.d/networking'' script. As with most ''/etc/init.d'' scripts, this script takes several options, such as ''start'', ''stop'', ''restart''. Note even if you stop networking, you would still have your lo interface. You can look at the code of the script to find out what it actually does. You can also stop or start individual interfaces by using the '''ifup''' and '''ifdown''' commands.

The network configuration files are stored in ''/etc/network''. ''/etc/network/interfaces'' contains the defaults for each interface. For xample, you could specify static IP, netmask, network, broadcast and default gateway for an interface here, but you should not need to edit this files in general. These default options can be changed with the '''ifconfig''' command. The ''/etc/network/if-down.d'' and ''/etc/network/if-up.d'' directories contain the scripts that are going to be executed when an interface is turned on or off. Of course, most modern Linux distributions have GUI tools for doing network configuration more easily, and you shouldn't need to change anything for the purposes of this course.

== Synchronizing Date and Time ==

In order to avoid setting your system's time manually at every daylight savings change, you can use a Network Time Server via the Network Time Protocol (NTP).

The NTP Daemon comes pre-installed on EC2 AMI instances.

{{RequiredInstructions|content=

=== Setting Timezone ===

Your server is probably not set to the correct timezone by default.

The timezone files are in the directory ''/usr/share/zoneinfo''. They are further organized within subdirectories grouped by region. For instance, Rome's time zone file is stored within ''/usr/share/zoneinfo/Europe''.

In order to set the time zone, simply copy the desired time zone file to our ''/etc'' directory as a new file named "localtime". For example, to set the the machine's system time to Rome's time zone, we would enter the command

<source lang="bash">
sudo cp /usr/share/zoneinfo/Europe/Rome /etc/localtime
</source>

''ntp'' uses ''/etc/ntp.conf'' configuration file to find the ostnames of remote time servers. The defaults here are probably fine.

}}

== Installing Software ==

The package management tool in Red Hat Enterprise Linux (and therefore also your Amazon EC2 instance) is '''rpm'''. If you have an ''rpm'' package, you can install it by

<source lang="bash">
$ rpm -i somepackage.rpm
$
</source>

This requires that ''somepackage.rpm'' be in your current directory, which means you will have to download the file yourself (or create it). It requires you to manually install any dependencies the package has.

=== Repository-Based Package Managers ===

A better alternative is to use a repository-based package manager. In RHEL, this is '''yum'''.

Before you install new software, you need to ensure that your local list of available packages is up-to-date. Run the following commands to perform this operation:

<source lang="bash>
yum check-update
</source>

After you have ensured that your package list is synced with the remote repository, you can start installing packages. To install a package, the following command:

<source lang="bash>
yum install <package-name>
</source>

If you get a permission denied error, try sudoing the command, like '''sudo apt-get install <xxxx>'''

For example, this is how you would install ''lynx'', a command-line web browser, in your RHEL Linux distribution:

<source lang="bash">
$ lynx --version # is lynx installed?
-bash: lynx: command not found
$ sudo yum check-update # sync package lists with the remote repositories
$ sudo yum install lynx # install the lynx package
Downloading Packages:
lynx-2.8.6-27.6.amzn1.i686.rpm | 1.8 MB 00:00
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Installing : lynx-2.8.6-27.6.amzn1.i686 1/1

Installed:
lynx.i686 0:2.8.6-27.6.amzn1

Complete!
$ lynx --version # test again to see if we have lynx installed
Lynx Version 2.8.6rel.5 (09 May 2007)
$
</source>

You can also search for available packages by name or by the name of a file that they install.

<source lang="bash">
$ yum search lynx # search for packages whose name contains lynx
======= N/S Matched: lynx =======
lynx.i686 : A text-based Web browser
$ yum provides lynx # search for packages that install a file or command named lynx
lynx-2.8.6-27.6.amzn1.i686 : A text-based Web browser
Repo : installed
Matched from:
Other : Provides-match: lynx
$
</source>

The list of repositories that '''yum''' searches is located at <code>/etc/yum.conf</code>. The list of repositories that '''apt''' searches is located at <code>/etc/apt/sources.list</code>.

{{RequiredInstructions|content=

=== Essential Packages ===

To save you some headaches later on, it is recommended that you install a few essential package bundles when you first create a new Linux instance. These packages include things like Make and a C compiler.

'''RHEL:'''

<source lang="bash">
$ sudo yum groupinstall "Development Tools"
$ sudo yum install kernel-devel kernel-headers
</source>

}}

=== Searching for Packages ===

You can search for packages by name, command, and so on. For example, to search for all packages relating to "lynx", you could do:

'''RHEL:'''
<source lang="bash">
$ yum search lynx
</source>

== Command Reference ==
Earlier, you saw that '''cat''' is a command that shows the contents of a file. Below is a list of other useful commands in Linux.

=== Navigation and File Management ===

* '''ls''' List file(s) in current working directory
** '''ll''' Shortcut to '''ls -l'''. List files with more details than '''ls'''. Only available in certain distributions
* '''cd''' Change working directory. Note: cd called without any arguments moves you to your home directory
* '''cp''' Copy a file
* '''mv''' Move or rename a file
* '''rm''' Remove a file
** '''rm -r''' Remove a directory and all files in it
* '''ln -s''' Create a symlink to a file
* '''mkdir''' Create a directory
* '''rmdir''' Remove a directory (directory must be empty; if it's not, use '''rm -r''')
* '''cat''' Display the contents of a file
* '''less''' Display the contents of a file, wait for the user at each page
* '''tail''' Display the last 20 lines of a file
** '''tail -f''' Display the last 20 lines of a file and then wait for changes, displaying them as they occur. Useful for monitoring log files.
* '''chown''' Change the owner of a file
* '''chgrp''' Change the group of a file
* '''chmod''' Change the security permissions of a file
* '''grep''' Display the lines of a file matching a user specified string
* '''diff''' Display the difference between two files

=== System Administration ===

* '''df''' Display free diskspace
* '''du''' Display disk usage
* '''free''' Display memory usage information
* '''date''' Display current time and date
* '''top''' Display the CPU and Memory usages of current processes
* '''ps''' Display current processes
* '''kill''' Terminate a running process
* '''killall''' Terminate the running process matching user specified criterias
* '''ping hostname''' Ping a host
* '''host''' Get the IP address of a host
* '''passwd''' Change the user password
* '''su user''' Switch to the privileges of another user
* '''shutdown''' Power off the computer
* '''reboot''' Reboot the computer
* '''clear''' Clear the terminal
* '''ifconfig''' Display/Configure a network device
* '''file''' Show the file type
* '''lsmod''' Display loaded kernel modules
* '''insmod''' Install a kernel module
* '''modprobe''' Load a kernel module (also load the dependencies)
* '''adduser''' Add a new user
* '''exit''' Exit from a shell
* '''lpr''' Print a file
* '''head''' Display lines at the beginning of a file
* '''tail''' Display lines at the end of a file
* '''pwd''' Display the name of the current working directory
* '''lsof''' Open files in the system
* '''netstat''' Statistics related to open sockets
* '''vmstat''' Reports information about processes, memory, IO, and CPU

=== File Editors ===
{{XKCD
|name=real_programmers
|id=378
}}

It is sometimes convenient to edit files using the command line. Three widely-used command line text editors are '''vi''', '''emacs''', and '''nano'''.

==== Vi ====

To edit a file using Vi, use the command '''vi'''. You will see something like this:

.
~
~
~
~
~
~
~
~
~
~
~
~
"myfile.txt" [New File]

To insert text into the file, press '''i''' once, then type away:

Hello World
~
~
~
~
~
~
~
~
~
~
~
~
-- INSERT --

To leave insert mode, press ESC.

To save your file, type the command ''':w''' and press Enter.

Hello World
~
~
~
~
~
~
~
~
~
~
~
~
"myfile.txt" [New] 1L, 12C written

To save and then close your file, type the command ''':wq'''. (To close the file without saving, use the command ''':q!''')

For more vi commands, see https://ss64.com/vi.html

==== Emacs ====

To edit a file using emacs, use the command '''emacs'''. You can start typing immediately:

Hello World

-uuu:**-F1 myfile.txt All L1 (Text)----------

To save a file, type '''C-x C-s''' (that means Control-X, then Control-S):

Hello World

-uuu:---F1 myfile.txt All L1 (Text)----------
Wrote /home/todd/myfile.txt

To quit emacs, type '''C-x C-c'''. (It will ask you whether or not to save the file if you've made changes.)

For more emacs commands, see http://souptonuts.sourceforge.net/chirico/emacs_ref.html

==== Nano ====

To edit a file using nano, use the command '''nano'''. You can start editing the file immediately:

GNU nano 2.0.6 File: myfile.txt

Hello World

^G Get Help ^O WriteOut ^R Read File ^Y Prev Page ^K Cut Text ^C Cur Pos
^X Exit ^J Justify ^W Where Is ^V Next Page ^U UnCut Text ^T To Spell

Nano tells you the commands you need right there so you don't have to always keep looking them up like with Vi and Emacs.

For more detail on Nano commands, see http://www.nano-editor.org/dist/v2.2/nano.html

== Linux Resources ==

[http://www.yolinux.com/TUTORIALS/LinuxTutorialSysAdmin.html Linux System Administration Tutorial]

[http://www-uxsup.csx.cam.ac.uk/pub/doc/suse/suse9.0/userguide-9.0/ch24.html Working with the Shell (SUSE Documentation)]

[http://en.wikipedia.org/wiki/Linux_kernel Linux Kernel]

[http://www.yolinux.com/TUTORIALS/LinuxTutorialAdvanced_vi.html VI Tutorial]

[http://students.cec.wustl.edu/~jg18/guide/ Making the Transition to Linux: A Guide to the Linux Command Line Interface for Students]

[[Category:Module 2]]

Linux

2024-01-23T03:34:37Z

Zach: /* Setting File Permissions */

[[wikipedia:Linux|Linux]] is an [[wikipedia:Free and open-source software|open-source]] operating system based on [[wikipedia:UNIX|UNIX]]. Linux is highly versatile and is used in a wide range of applications. [[wikipedia:Desktop Linux|Desktop Linux]] is Linux with a GUI (like Microsoft Windows or Mac OS X); Desktop Linux is popular in niche markets, and it is used widely in developing countries.

Linux is the most widely used operating system for web servers. In CSE330, we will be interacting with Linux from the command line. This article covers the tools you need to make the best use of Linux.

== Linux Distributions ==

The open-source community is responsible for the development of many different [[wikipedia:Linux distributions|distributions]] of Linux. Distributions, or ''distros'', are different "flavors" of the Linux operating system with different objectives.

There are hundreds of distributions of Linux. Three of the main branches are [[wikipedia:Debian|Debian]], [[wikipedia:SuSE|SuSE]] (based on [[wikipedia:Slackware|Slackware]]), and [[wikipedia:Red Hat Enterprise Linux|Red Hat Enterprise Linux (RHEL)]].

For CSE 330, you'll be using a version of Red Hat Enterprise Linux.

=== Red Hat Enterprise Linux (RHEL) ===

Red Hat Enterprise Linux, or '''RHEL''', was first introduced in 1994. RHEL is known for being a good choice for enterprises that wish to use Linux as their primary OS. RHEL has an abundance of administration tools.

The Linux Lab in Lopata Hall uses '''[[wikipedia:Fedora Linux|Fedora Linux]]''', a desktop distribution based on RHEL. '''[[wikipedia:CentOS|CentOS]]''' is a popular RHEL derivative that is widely used in web servers. The '''[[wikipedia:Amazon Machine Image|Amazon EC2 Linux AMI]]''' is derived from CentOS, making it a distribution in the RHEL family.

=== Special Note: Linux Kernel and Modules ===
{{XKCD
|id=619
|name=supported_features
}}

What separates Linux from other Unix variants is its kernel. The kernel is the most important component of the operating system and is responsible for scheduling processes, providing access to the hardware devices, allocating memory to the programs, and so on.

The Linux kernel uses both monolithic and modular approaches. A monolithic kernel is a single program that contains all the code so any addition to kernel (such as code to access a driver) requires recompiling the code. A monolithic kernel is usually a little faster and could have a smaller size since only the absolutely necessary code is there. The modular kernel, on the other hand, enables dynamic loading and unloading of kernel code, called ''modules''. Typical modules include device drivers. Thanks to this modular approach, Linux seldom requires a reboot after installing a new device.

== Bash ==

In this course, you will be interacting with Linux primarily on the command line. The most widespread ''command line language'' is '''Bash'''. This is described in more detail in [[Bash|the Bash guide]].

== Files and Permissions ==

At the core of a Unix-based operating system is a directory structure with files and permissions.

=== Filesystem Hierarchy ===

The root directory of Linux contains a dozen or so subdirectories, each with a specific purpose:

* '''/bin''' contains [[Bash#Running Programs|binaries]] used by all users
* '''/sbin''' contains system binaries typically used only by the system administrator
* '''/lib''' contains libraries for the binaries found in '''/bin''' and '''/sbin'''
* '''/etc''' contains configuration files
** '''/etc/yum.conf''' Configuration file for yum
** '''/etc/yum/yum.repos.d''' Directory containing .repo files for online repositories
** '''/etc/crontab''' System-wide crontab file
** '''/etc/fstab''' Information about default partitions to be mounted
** '''/etc/group''' List of groups in the system
** '''/etc/hosts''' List of IP addresses with their names
** '''/etc/inittab''' What to do at each run-level
** '''/etc/inetd.conf''' Configuration file for some internet services (replaced by xinetd.* in most systems)
** '''/etc/modules.conf''' Module information for the boot
** '''/etc/motd''' Message to be seen at the login prompt
** '''/etc/passwd''' User information
** '''/etc/profile''' System level initial file for sh and its derivatives
** '''/etc/shadow''' User passwords
* '''/dev''' contains device files
* '''/proc''' contains information on currently running processes
* '''/var''' contains files whose contents is expected to change
** '''/var/log''' contains system log files
*** '''/var/log/messages''' System/Kernel messages
*** '''/var/log/syslog''' System log (mostly for Daemons)
*** '''/var/log/wtmp'''' User access log (binary)
*** '''/var/log/dmesg''' Boot-up messages
*** '''/var/log/auth.log''' Authorization logs
** '''/var/lib''' contains packages and database files
** '''/var/spool''' contains print queues
* '''/tmp''' contains temporary files that are deleted at system reboot
* '''/usr''' contains user programs
** '''/usr/bin''' contains binaries for user programs
** '''/usr/sbin''' contains binaries for system administrators
** '''/usr/lib''' contains libraries for '''/usr/bin''' and '''/usr/sbin'''
** '''/usr/local''' contains programs that you install from source
* '''/home''' contains users' home directories
* '''/root''' is root's home directory
* '''/boot''' contains boot loader files (do not touch unless you know what you are doing!)
* '''/opt''' contains optional add-on applications
* '''/mnt''' is where system administrators can mount filesystems
* '''/media''' contains links to removable media devices (for example, CDs)
* '''/srv''' contains site-specific data which are served by the system

For more information, see the Wikipedia article on the [[wikipedia:Filesystem Hierarchy Standard|Filesystem Hierarchy Standard]].

=== File Permissions ===

Every file in Linux has permissions that define which users can Read, Write, and Execute it. Every file has an owner and a group. The permissions for a file are set on three levels: User (owner), Group, and Other.

==== Symbolic Notation ====

When you view the permissions of a file in Linux, they will most often be displayed in ''symbolic notation''. Symbolic notation consists of 10 characters: the first defines the file type, and then there are three characters each for User, Group, and Other permissions.

* <code>-r--r--r--</code> is a normal file that is readable by all users but writable or executable by no one.
* <code>-rwxr-xr-x</code> is a normal file that is readable and executable by everyone but only writable by User (the file's owner). This is the most common permission set.

==== Viewing File Permissions ====

To view the permissions of all files in a certain directory, run the binary <code>ls -l</code> in Bash:

<source lang="bash">
$ ls -l # displays a list of all files in a directory with their permissions in symbolic notation
total 16
lrwxr-xr-x 1 sffc wheel 6 Aug 9 09:13 link -> myfile.txt
-rwxr--r-- 1 sffc wheel 12 Aug 9 09:13 myfile.txt
$ ls -l myfile.txt # displays the permissions of only myfile.txt
-rwxr--r-- 1 sffc wheel 12 Aug 9 09:13 myfile.txt
$
</source>

==== Setting File Permissions ====

Linux comes with several useful binaries for setting file permissions.

* '''chmod''' is used for setting permissions
* '''chown''' is used for setting a file's owner
* '''chgrp''' is used for setting a file's group

Some examples are shown below.

<source lang="bash">
$ chmod a+x myfile.txt # turns on the Execute option for all users
$ chmod o-w myfile.txt # turns off the Write option for Other users
$ chmod u+wx-r myfile.txt # turns on the Write and Execute options for User (the file's owner) and also turns off the Read option for User
$ chown todd myfile.txt # sets the owner of myfile.txt to the user todd. Note: First comes the user, then comes the filename: not the other way around!
$ chgrp staff myfile.txt # sets the group of myfile.txt to usergroup staff
$
</source>

For more information, see https://www.linux.com/training-tutorials/how-manage-file-and-folder-permissions-linux/

=== The . and .. Directories ===

The '''.''' directory is a reference to the current directory. The '''..''' directory brings you one level up in the filesystem.

=== Symbolic Links ===

A [[wikipedia:Symbolic link|symbolic link]], or ''symlink'', is basically a link from one spot in the filesystem to another. You can think of them like ''aliases'' in Mac OS X. To create a a symlink, use the ''ln -s'' command:

<source lang="bash">
$ ln -s /path/to/file.txt /path/to/link # creates a symlink to file.txt at /path/to/link

# Example:
$ ln -s /home/todd/instructions.doc /var/www/public_html/classes/instructions.doc # creates a symlink in the web server to instructions.doc
$ vi /var/www/public_html/classes/instructions.doc # changes to the symbolic link will be reflected in the original file
$
</source>

== User Management ==

=== Adding a New User ===

To create a new user, use the '''useradd''' command. Then, set a password for that user using the '''passwd''' command:

<source lang="bash">
$ sudo useradd -r -m -c "<My Full Name>" <myUserName>
$ sudo passwd <myUserName>
Enter new UNIX password:
Retype new UNIX password:
$
</source>

'''Note:''' If Linux doesn't like your command, first check your syntax and then try it with single quotes around your full name and password instead of double quotes. If this works it is likely that you used some special characters in your password such as '!', which serves a special purpose in the shell. This is because Linux takes parameters enclosed in double quotes as one string, but still looks for special characters inside of the string and acts in accordance with their normal meaning. On the other hand, single quotes tell the shell to ignore any special characters within.

=== Adding a User to the Sudoers List ===

For security reasons, you should never SSH into your server as the root user. Instead, you should use a normal user to whom you give '''sudo''' privileges. (For more detail on sudo, see [[Linux#SUDO|the Linux guide]].)

To give a user sudo privileges, use the command '''visudo''', which opens up the SUDO configuration file in the system's default text editor. (Never edit the file ''/etc/sudoers'' directly!) SUDO users are specified using lines similar to

alice ALL=(ALL) ALL

Add that line immediately below the line defining '''root''':

root ALL=(ALL) ALL

When you are finished, save and close the file. For more information on editing files on the command line, refer to [[#File Editors|the section about File Editors in this guide]].


In this case, '''alice''' will be able to run any command with SUDO privileges on the computer. For more detail on SUDO configuration, see http://www.linuxhelp.net/guides/sudo/

'''Note:''' The visudo utility is especially useful because it not only locks the sudoers file against simultaneous edits, but it also checks the file for syntax errors when you quit. Therefore, it is always recommended that you use visudo to change the sudoers list rather than attempting to modify it directly.

== Networking ==

In Linux, you can see your network information by typing '''ifconfig'''. This command shows the status information of each network interface, including the IP address you will need to remotely connect to your instance. The interface ''lo'' is the special loopback interface with IP address 127.0.0.1. This refers to your local machine and any connection from your machine to your machine goes through this pseudo-interface. Typical network interfaces include ''eth0'', ''eth1'',..., ''wlan0'', etc. Ethernet cards are represented with ''ethX''. In the past, most wireless cards showed up as ''wlanX'', but it is also common now for them to be represented with ''ethX'' names. ''ifconfig'' also gives information such as hardware address (MAC), netmask, and broadcast addresses.

You can start or stop networking by calling ''/etc/init.d/networking'' script. As with most ''/etc/init.d'' scripts, this script takes several options, such as ''start'', ''stop'', ''restart''. Note even if you stop networking, you would still have your lo interface. You can look at the code of the script to find out what it actually does. You can also stop or start individual interfaces by using the '''ifup''' and '''ifdown''' commands.

The network configuration files are stored in ''/etc/network''. ''/etc/network/interfaces'' contains the defaults for each interface. For xample, you could specify static IP, netmask, network, broadcast and default gateway for an interface here, but you should not need to edit this files in general. These default options can be changed with the '''ifconfig''' command. The ''/etc/network/if-down.d'' and ''/etc/network/if-up.d'' directories contain the scripts that are going to be executed when an interface is turned on or off. Of course, most modern Linux distributions have GUI tools for doing network configuration more easily, and you shouldn't need to change anything for the purposes of this course.

== Synchronizing Date and Time ==

In order to avoid setting your system's time manually at every daylight savings change, you can use a Network Time Server via the Network Time Protocol (NTP).

The NTP Daemon comes pre-installed on EC2 AMI instances.

{{RequiredInstructions|content=

=== Setting Timezone ===

Your server is probably not set to the correct timezone by default.

The timezone files are in the directory ''/usr/share/zoneinfo''. They are further organized within subdirectories grouped by region. For instance, Rome's time zone file is stored within ''/usr/share/zoneinfo/Europe''.

In order to set the time zone, simply copy the desired time zone file to our ''/etc'' directory as a new file named "localtime". For example, to set the the machine's system time to Rome's time zone, we would enter the command

<source lang="bash">
sudo cp /usr/share/zoneinfo/Europe/Rome /etc/localtime
</source>

''ntp'' uses ''/etc/ntp.conf'' configuration file to find the ostnames of remote time servers. The defaults here are probably fine.

}}

== Installing Software ==

The package management tool in Red Hat Enterprise Linux (and therefore also your Amazon EC2 instance) is '''rpm'''. If you have an ''rpm'' package, you can install it by

<source lang="bash">
$ rpm -i somepackage.rpm
$
</source>

This requires that ''somepackage.rpm'' be in your current directory, which means you will have to download the file yourself (or create it). It requires you to manually install any dependencies the package has.

=== Repository-Based Package Managers ===

A better alternative is to use a repository-based package manager. In RHEL, this is '''yum'''.

Before you install new software, you need to ensure that your local list of available packages is up-to-date. Run the following commands to perform this operation:

<source lang="bash>
yum check-update
</source>

After you have ensured that your package list is synced with the remote repository, you can start installing packages. To install a package, the following command:

<source lang="bash>
yum install <package-name>
</source>

If you get a permission denied error, try sudoing the command, like '''sudo apt-get install <xxxx>'''

For example, this is how you would install ''lynx'', a command-line web browser, in your RHEL Linux distribution:

<source lang="bash">
$ lynx --version # is lynx installed?
-bash: lynx: command not found
$ sudo yum check-update # sync package lists with the remote repositories
$ sudo yum install lynx # install the lynx package
Downloading Packages:
lynx-2.8.6-27.6.amzn1.i686.rpm | 1.8 MB 00:00
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Installing : lynx-2.8.6-27.6.amzn1.i686 1/1

Installed:
lynx.i686 0:2.8.6-27.6.amzn1

Complete!
$ lynx --version # test again to see if we have lynx installed
Lynx Version 2.8.6rel.5 (09 May 2007)
$
</source>

You can also search for available packages by name or by the name of a file that they install.

<source lang="bash">
$ yum search lynx # search for packages whose name contains lynx
======= N/S Matched: lynx =======
lynx.i686 : A text-based Web browser
$ yum provides lynx # search for packages that install a file or command named lynx
lynx-2.8.6-27.6.amzn1.i686 : A text-based Web browser
Repo : installed
Matched from:
Other : Provides-match: lynx
$
</source>

The list of repositories that '''yum''' searches is located at <code>/etc/yum.conf</code>. The list of repositories that '''apt''' searches is located at <code>/etc/apt/sources.list</code>.

{{RequiredInstructions|content=

=== Essential Packages ===

To save you some headaches later on, it is recommended that you install a few essential package bundles when you first create a new Linux instance. These packages include things like Make and a C compiler.

'''RHEL:'''

<source lang="bash">
$ sudo yum groupinstall "Development Tools"
$ sudo yum install kernel-devel kernel-headers
</source>

}}

=== Searching for Packages ===

You can search for packages by name, command, and so on. For example, to search for all packages relating to "lynx", you could do:

'''RHEL:'''
<source lang="bash">
$ yum search lynx
</source>

== Command Reference ==
Earlier, you saw that '''cat''' is a command that shows the contents of a file. Below is a list of other useful commands in Linux.

=== Navigation and File Management ===

* '''ls''' List file(s) in current working directory
** '''ll''' Shortcut to '''ls -l'''. List files with more details than '''ls'''. Only available in certain distributions
* '''cd''' Change working directory. Note: cd called without any arguments moves you to your home directory
* '''cp''' Copy a file
* '''mv''' Move or rename a file
* '''rm''' Remove a file
** '''rm -r''' Remove a directory and all files in it
* '''ln -s''' Create a symlink to a file
* '''mkdir''' Create a directory
* '''rmdir''' Remove a directory (directory must be empty; if it's not, use '''rm -r''')
* '''cat''' Display the contents of a file
* '''less''' Display the contents of a file, wait for the user at each page
* '''tail''' Display the last 20 lines of a file
** '''tail -f''' Display the last 20 lines of a file and then wait for changes, displaying them as they occur. Useful for monitoring log files.
* '''chown''' Change the owner of a file
* '''chgrp''' Change the group of a file
* '''chmod''' Change the security permissions of a file
* '''grep''' Display the lines of a file matching a user specified string
* '''diff''' Display the difference between two files

=== System Administration ===

* '''df''' Display free diskspace
* '''du''' Display disk usage
* '''free''' Display memory usage information
* '''date''' Display current time and date
* '''top''' Display the CPU and Memory usages of current processes
* '''ps''' Display current processes
* '''kill''' Terminate a running process
* '''killall''' Terminate the running process matching user specified criterias
* '''ping hostname''' Ping a host
* '''host''' Get the IP address of a host
* '''passwd''' Change the user password
* '''su user''' Switch to the privileges of another user
* '''shutdown''' Power off the computer
* '''reboot''' Reboot the computer
* '''clear''' Clear the terminal
* '''ifconfig''' Display/Configure a network device
* '''file''' Show the file type
* '''lsmod''' Display loaded kernel modules
* '''insmod''' Install a kernel module
* '''modprobe''' Load a kernel module (also load the dependencies)
* '''adduser''' Add a new user
* '''exit''' Exit from a shell
* '''lpr''' Print a file
* '''head''' Display lines at the beginning of a file
* '''tail''' Display lines at the end of a file
* '''pwd''' Display the name of the current working directory
* '''lsof''' Open files in the system
* '''netstat''' Statistics related to open sockets
* '''vmstat''' Reports information about processes, memory, IO, and CPU

=== File Editors ===
{{XKCD
|name=real_programmers
|id=378
}}

It is sometimes convenient to edit files using the command line. Three widely-used command line text editors are '''vi''', '''emacs''', and '''nano'''.

==== Vi ====

To edit a file using Vi, use the command '''vi'''. You will see something like this:

.
~
~
~
~
~
~
~
~
~
~
~
~
"myfile.txt" [New File]

To insert text into the file, press '''i''' once, then type away:

Hello World
~
~
~
~
~
~
~
~
~
~
~
~
-- INSERT --

To leave insert mode, press ESC.

To save your file, type the command ''':w''' and press Enter.

Hello World
~
~
~
~
~
~
~
~
~
~
~
~
"myfile.txt" [New] 1L, 12C written

To save and then close your file, type the command ''':wq'''. (To close the file without saving, use the command ''':q!''')

For more vi commands, see https://ss64.com/vi.html

==== Emacs ====

To edit a file using emacs, use the command '''emacs'''. You can start typing immediately:

Hello World

-uuu:**-F1 myfile.txt All L1 (Text)----------

To save a file, type '''C-x C-s''' (that means Control-X, then Control-S):

Hello World

-uuu:---F1 myfile.txt All L1 (Text)----------
Wrote /home/todd/myfile.txt

To quit emacs, type '''C-x C-c'''. (It will ask you whether or not to save the file if you've made changes.)

For more emacs commands, see http://souptonuts.sourceforge.net/chirico/emacs_ref.html

==== Nano ====

To edit a file using nano, use the command '''nano'''. You can start editing the file immediately:

GNU nano 2.0.6 File: myfile.txt

Hello World

^G Get Help ^O WriteOut ^R Read File ^Y Prev Page ^K Cut Text ^C Cur Pos
^X Exit ^J Justify ^W Where Is ^V Next Page ^U UnCut Text ^T To Spell

Nano tells you the commands you need right there so you don't have to always keep looking them up like with Vi and Emacs.

For more detail on Nano commands, see http://www.nano-editor.org/dist/v2.2/nano.html

== Linux Resources ==

[http://www.yolinux.com/TUTORIALS/LinuxTutorialSysAdmin.html Linux System Administration Tutorial]

[http://www-uxsup.csx.cam.ac.uk/pub/doc/suse/suse9.0/userguide-9.0/ch24.html Working with the Shell (SUSE Documentation)]

[http://en.wikipedia.org/wiki/Linux_kernel Linux Kernel]

[http://www.yolinux.com/TUTORIALS/LinuxTutorialAdvanced_vi.html VI Tutorial]

[http://students.cec.wustl.edu/~jg18/guide/ Making the Transition to Linux: A Guide to the Linux Command Line Interface for Students]

[[Category:Module 2]]

Module 2

2024-01-22T22:30:05Z

Zach: /* Calculator */

__NOTOC__
Module 2 introduces you to Linux, a command-line environment, the Apache web server, and PHP. You will create and configure your own cloud instance, install Apache and related software, and then form groups to make a simple file sharing site.

<span style="font-size:1.3em; line-height:1.2em; color:blue;">'''DO NOT WAIT UNTIL THE LAST MINUTE TO START ON THIS (or any) MODULE!''' The most common reason students perform poorly in this class is procrastination. You have been warned.</span>

== Using the Wiki ==
{{RequiredInstructions|content=
Text enclosed by <syntaxhighlight lang="bash" inline><</syntaxhighlight> and <syntaxhighlight lang="bash" inline>></syntaxhighlight> should be replaced by content unique to you.

'''Example'''

<source lang="bash">
$ sudo useradd -r -m -c "<My Full Name>" <usernameHere>
</source>
becomes
<source lang="bash">
$ sudo useradd -r -m -c "Zach Cohn" zcohn
</source>
}}

== Reading ==

The following articles on the online class wiki textbook contain information that will help you complete the assignments.

* [[Linux]]
* [[Bash]]
* [[Amazon Web Services]]
* [[SSH]]
* [[Apache]]
* [[PHP]]
* [[Workflow]]
* [[VS Code]]
* [[Web Application Security, Part 1]]
* [[FAQ - Mod 2]]

== Individual Assignments ==

{{RequiredInstructions|content=

<span style="color:red">There are a lot of wiki pages that you need to read in order to complete the individual portion of Module 2. The wiki pages contain an assortment of required steps, optional steps, and information. To make things easier, we have highlighted the ''required steps'' with a red bar, like you see on the left of this message. Do not skip anything marked with a red bar, or else you will not have a perfectly configured instance.</span>

}}

=== Learn About Linux ===

Linux is an open-source operating system based on UNIX. Linux is highly versatile and is used in a wide range of applications.

If you are not a Linux guru, read the Linux guide to get started: [[Linux]]

=== Windows Users Only: Enable Windows Subsystem for Linux (WSL) or Install Cygwin ===

You will need WSL or Cygwin in order to log into and configure your EC2 instance. Cygwin is already installed on the CEC machines. If you're using Windows 10, you can use WSL, which is an officially-supported way of accessing a Linux terminal on Windows. If not, you'll have to use Cygwin. Instructions for installing either on your personal machine are located in [[Workflow#Cygwin|the Workflow guide]].

Mac OS X users should located the "Terminal" application, which is installed by default; you will need this in the coming steps.

=== Create an AWS EC2 Instance ===

Once you understand [[Linux]], you need to set up your Amazon EC2 Instance. Use the AWS article to guide you through the process: [[Amazon Web Services]]

=== Configure SSH ===

Follow the instructions in [[SSH#SSH Configuration|the SSH guide]] to do the following:

# Create your own SSH key pair
# Log in as the default user to your EC2 instance
# Create a new user
# Give the new user your SSH public key

From now on, when you need to log into your EC2 instance as an administrator, just pop up a terminal and run:

$ ssh <your-instance-username>@<ec2-xx-xx-xx-xx.compute-1.amazonaws.com>

To save from typing this command in every time you want to access your server, you can set an '''alias''' in your terminal.

In the '''Mac''' terminal, vim ~/.zshrc and input the following command:
$ alias aws='ssh <your-instance-username>@<ec2-xx-xx-xx-xx.compute-1.amazonaws.com>'
Now, whenever you type aws in your terminal, it will type your ssh command.

In '''Cygwin''', vim ~/.bash_profile and input the following:
export aws="ssh <your-instance-username>@<ec2-xx-xx-xx-xx.compute-1.amazonaws.com>"
And you run this command by typing '''$aws'''
Also note that you may have to restart your terminal for this to take effect, or put the command in your ~/.bashrc file

=== Install Essential Packages ===

Recall that software installation in Linux instances is usually handled through a package manager, like '''yum'''. For more information, refer to [[Linux#Repository-Based Package Managers|the Linux guide]].

To make sure everything is working properly, '''install your first package(s) according to the ''Essential Packages'' subsection in the Linux guide.'''

=== Set the Timezone ===

'''Set your server to use US Central time (America/Chicago).''' Instructions are in the Linux guide: [[Linux#Synchronizing Date and Time]]

=== Set Up the Apache Web Server ===

You need to '''install the Apache web server on your EC2 instance'''. Refer to [[Apache|the Apache guide]].

In order for your web server to be accessible, you need to '''open up Port 80 on your EC2 instance'''. Instructions are in the AWS guide: [[Amazon Web Services#Enabling Web Access to your EC2 Instance]]

'''Enable the UserDir module in Apache.''' Instructions are in the Apache guide: [[Apache#The UserDir Module]]

=== Install PHP ===

You need to '''install PHP on your EC2 server'''. Instructions are at the top of the PHP guide: [[PHP]]

You need to '''configure PHP to show errors'''. Instructions are immediately below installation in the PHP guide.

=== Learn about the PHP Language ===

Before you can continue, you will need to make yourself acquainted with PHP language components. Read the PHP guide for an overview: [[PHP#PHP Language Components]]

=== Set Up your Workspace ===

You are now ready to set up the workspace that you will be using for the rest of the semester. Follow the instructions in [[Workflow]] to:

# Make an account on GitHub
# Setup a repo using GitHub Classroom
# Clone your CSE 330 repo from GitHub
# Install a text editor such as '''VSCode''' to write your assignments with
# Either install an FTP client, or use SFTP through command line, to transfer files from your local machine to your instance

=== Calculator ===

'''Make a calculator using PHP and an HTML form.''' The form should have two number inputs. The form should submit a GET request either back to the same page or to a different results page. The calculator should support addition, subtraction, multiplication, and division by means of a radio button group on the page. (Consider edge cases!)

=== Git Tutorial ===

Complete all of the levels of the Main and Remote sections of [https://learngitbranching.js.org/ this Git tutorial]. After you finish the tutorial, take two screenshots showing the completion of all of the levels of the Main and Remote sections. Place these two screenshots in your Module 2 Individual repo.

== Group Project ==

You will work in pairs (that means you and one other person) on this project.

<span style="font-size:1.3em; line-height:1.2em; color:blue;">In this module, and all future modules, the group portion builds off of material you learn in the individual portion. You should therefore complete the individual portions prior to working on the group portions. (Trust us: it will make your life easier!)</span>

'''Important Reminder:''' frequently commit your work to your repository as a backup!

=== Simple File Sharing Site ===

You will be making a simple file sharing site that supports uploading, viewing, and deleting files associated with various users. Details:

* You should have a file named '''users.txt''' stored in a secure place on your filesystem. It should have at least three usernames, with one username per line.
*: Passwords add an extra layer of complexity (encryption) that we will cover in Module 3. You will not earn credit for implementing passwords as your creative portion for this module.
* Users of the file sharing site should be able to enter their username and then log in.
*: You may implement logins using session variables, or you may implement them using a GET parameter that is passed between pages. You will need to always use sessions starting in Module 3.
* Users should see a list of all files associated with their username after the log in.
* Users should be able to view, upload, and delete files associated with their username.
* The URI should NOT reveal the internal file structure of your web site.
** Example of what not to do: http://www.example.com/download.php?file=/home/paul/file.txt
** Another bad example: http://www.example.com/~user/file.txt
** Good example: http://www.example.com/view.php?user=music&name=file.txt
*: In general, don't reveal any internal information about the site.
*: '''Food for thought:''' Given that Apache processes (with the exception of the main process) do not run as root, but rather as the user apache, how can you allow both your user account and the php scripts to read and write the
*:necessary files?
*:: ''Hint:'' You shouldn't be storing the user-uploaded files in public_html (that is only the place for your own web-page files),
*::''Hint number 2:'' you will need to use chown to change permissions to let user apache have permissions on the user-files (that '''are not''' in the public_html folder). If you don't know what this does or how to do this, look at the [[Apache]] page
* You should customize your file sharing site by implementing an additional feature or two as part of the '''creative portion'''.

You will probably find the PHP guide on this wiki to be helpful: [[PHP#Other PHP Tips]]

=== Web Security and Validation ===

Your project needs to demonstrate that thought was put into web security and best practice. For more information, see this week's Web Application Security guide: [[Web Application Security, Part 1]]

In particular:

* '''Your application needs to follow the conventions of FIEO (Filter Input and Escape Output)'''. Things you should filter include file names, usernames, and so on.
* '''Your project must pass the W3C Validator with no errors or warnings'''. For more information, see the HTML and CSS guide: [[HTML and CSS#Validation]]

== Grading ==

We will be grading the following aspects of your work. There are 120 points total.

'''Assignments (including code) must be committed to GitHub by the end of class on the due date. Commit early and often. Failing to commit by the end of class on the due date will result in a 0.

_____________

# '''Cloud Instance and Web Server (30 Points):'''
#* You can SSH into your instance using your own custom username and SSH key. Put a screenshot in your repository of your terminal SSH-ing into your instance (15 points)
#* Your custom username is able to perform SUDO commands. Put a screenshot in your repository of you successfully executing a command with sudo (5 points)
#* The time zone is correct on your instance. Put a screenshot in your repository of the results of the `date` command (5 points)
#* The UserDir Apache module is working. Place a file in /home/<username>/public_html and link to the file in your README.md (5 points)
# '''PHP Calculator (15 Points):'''
#* The calculator is able to perform all four functions with floating point numbers (2 points each, 8 points in total)
#* Each function is given as a radio button choice (4 points)
#* The calculator page passes the W3C HTML validator (3 points)
#* '''''Make sure you have a README.md file in your individual repo with the link to your calculator on your instance.''''' See the Creative Portion section for more information about .md files.
# '''Git Tutorial (20 Points):'''
#* Two screenshots showing the completion of all of the levels of the Main and Remote sections placed in your individual assignment repository.
# '''File Sharing Site (40 Points):'''
#* '''''File Management (25 Points):'''''
#** Users should not be able to see any files until they enter a username and log in (4 points)
#**: ''Remember that users.txt should be stored in a secure location on your filesystem. That is, '''you should not be able to type any URL into your browser and see the raw users.txt file!'''''
#** Users can see a list of all files they have uploaded (4 points)
#** Users can open files they have previously uploaded (5 points)
#**: ''Note: Users should be able to open not only plain text files but also other file formats: images, spreadsheets, etc.''
#** Users can upload files (4 points)
#**: ''Note: Like users.txt, uploaded files should be stored in a secure location on your filesystem. That is, '''do not keep your uploads directory underneath a directory served by Apache!'''''
#** Users can delete files. If a file is "deleted", it should actually be removed from the filesystem (4 points)
#** The directory structure is hidden. Users should not be able to access or view files by manipulating a URL. (2 points)
#** Users can log out (2 points)
#**: ''Note: If using session variables, you must ''actually'' log out the user by destroying their session; i.e., don't just redirect them to the login screen.''
#* '''''Best Practices (10 Points):'''''
#** Code is well formatted and easy to read, with proper commenting (4 points)
#** The site follows the FIEO philosophy (3 points)
#** All pages pass the W3C validator (3 points)
#* '''''Usability (5 Points):'''''
#** Site is intuitive to use and navigate (4 points)
#** Site is visually appealing (1 point)
# '''Creative Portion (15 Points)''' (see below)
#* '''''Make sure you have a README.md file in your group repo with the following:'''''
#** An .md file is a plain text file called a markdown file [https://guides.github.com/features/mastering-markdown/ ] .
#***The contents of your README.md file are shown on your repo, making it easier for us to grade
#*** You may style the page with bolding, hyperlings, images, block quotes and lists
#*** [http://dillinger.io/ This online markdown editor] makes it easy to learn this format
#** The link to your file sharing site. '''''We need this to grade your work.'''''
#** A brief description of what you did for your creative portion
#** Any additional login details needed for the TA

=== Creative Portion ===

This module and all future modules will require that you invest some time into creating additional features for your group project. Plan to invest at the very least 60 minutes of your time into the creative portion. In the above rubric, viewing files is worth five and uploading is worth four. The creative portion is worth fifteen points; it should be a very sizable part of your project.

* The creative portion is an opportunity for you to learn the material of your own interest.
* You will not earn credit for a creative-portion feature that simply rehashes something you've already done in a previous module.

If you need ideas for a creative portion, or if you want to know whether or not your creative portion idea is "hard enough", ask a TA.

[[Category:Module 2]]
[[Category:Modules]]

PHP and MySQL

2023-09-26T00:08:58Z

Zach: /* Connecting to a MySQL Database in PHP */

MySQL is great, but it's not useful for web applications without a way to integrate it with your application logic. This article discusses how to query MySQL from within PHP.

This article assumes that you are comfortable with the content in the following guides. If you are not, you should read the guides listed below first.

* [[PHP]]
* [[Introduction to MySQL]]
* [[MySQL Schema and State]]

{{XKCD
|id=970
|name=the_important_field
}}

== PHP Libraries for Database Manipulation ==

PHP comes with many different ways of querying databases. This guide instructs you on how to use '''MySQL Improved Prepared Statements''', which is a cleaner, more elegant way to perform queries on your database.

== Connecting to a MySQL Database in PHP ==

To connect to a database using the MySQL Improved interface, use the mysqli command:

<source lang="php">
<?php
// Content of database.php

$mysqli = new mysqli('localhost', '<username>', '<password>', '<databasename>');

if($mysqli->connect_errno) {
printf("Connection Failed: %s\n", $mysqli->connect_error);
exit;
}
?>
</source>

You need to run this code before you can perform ''any'' other query on your database. Many PHP web developers put this code in a separate file and '''require''' it into other files that require database access.

You should create a new MySQL user with minimal privileges just for PHP to use.

== Performing Queries that Require Parameters ==

When you want to INSERT or UPDATE an entry in your database, you probably need to supply some value from PHP for MySQL to interpret. This section discusses ''parameter binding'', which is how you can do this sort of thing.

The first type of prepared query we will discuss are INSERT queries.

Suppose you had a form that submitted the following information:

* '''first''', the user's first name
* '''last''', the user's last name
* '''dept''', the user's department

You could insert data received from the form into the employees database using the following PHP code:

<source lang="php">
<?php
require 'database.php';

$first = $_POST['first'];
$last = $_POST['last'];
$dept = $_POST['dept'];

$stmt = $mysqli->prepare("insert into employees (first_name, last_name, department) values (?, ?, ?)");
if(!$stmt){
printf("Query Prep Failed: %s\n", $mysqli->error);
exit;
}

$stmt->bind_param('sss', $first, $last, $dept);

$stmt->execute();

$stmt->close();

?>
</source>

The query that will end up being executed by the above code is:

<source lang="mysql">
insert into employees (first_name, last_name, department) values ('John', 'Deere', 'EECE')
</source>

Notice how the MySQLi interface uses object-oriented PHP. Refer to [[PHP#Object-Oriented Programming|the PHP guide]] if you need a refresher on object-oriented PHP.

=== bind_param() ===

The most tricky line in the above example is the '''$stmt->bind_param()''' line. Here's how it works.

The first parameter defines the data types of data to be inserted into the query. It is a string with one character per data type. The possible data types are:

* '''i''' - Integer
* '''d''' - Decimal
* '''s''' - String
* '''b''' - Blob

Note that data types in queries are ''not'' the same as data types in schemas. For instance, when saving a date into MySQL, you need to use a string representation of the data (except for timestamp, which uses an integer), so you would use the '''s''' option.

The remaining parameters define values associated with each of the question marks in your query.

In the example code in the previous section, there are three question marks in the query, all of which are strings, and so the first argument to '''$stmt->bind_param()''' is 'sss'. The remaining three parameters is the information itself to be "injected" into the query.

== Performing Queries that Return Data ==

When you perform a successful SELECT query, MySQL always returns a table with the results of the query. This section discusses ''result binding'', which is how you can load the result table into PHP.

Consider the following example:

<source lang="php">
<?php
require 'database.php';

$stmt = $mysqli->prepare("select first_name, last_name from employees order by last_name");
if(!$stmt){
printf("Query Prep Failed: %s\n", $mysqli->error);
exit;
}

$stmt->execute();

$stmt->bind_result($first, $last);

echo "<ul>\n";
while($stmt->fetch()){
printf("\t<li>%s %s</li>\n",
htmlspecialchars($first),
htmlspecialchars($last)
);
}
echo "</ul>\n";

$stmt->close();
?>
</source>

Here, we use '''$stmt->bind_result()''' instead of '''$stmt->bind_param()''' because we are interested in the output from, not the input to, the query.

=== bind_result() and fetch() ===

Here, our query returns a table with two columns: first_name and last_name. So, we need to pass two parameters into '''$stmt->bind_result()''': one to store the first name, and one to store the last name.

After we call '''$stmt->bind_result()''', we call '''$stmt->fetch()'''. '''$stmt->fetch()''' stores the values from the next row in the result to the variables you specified in '''$stmt->bind_result()'''. '''$stmt->fetch()''' will continue binding rows into the variables until there are no rows left in the result set, at which time it will return null, thereby leaving the ''while'' loop.

'''Note:''' Use very unique variable names inside '''$stmt->bind_result()''', because whatever variables you use will be overwritten. This includes session variables, which are sometimes linked to the local variable of the same name.

=== get_result() and fetch_assoc() ===

If you do not want to bind the results from your query to variables, there is an alternative option that returns the results in arrays.

Consider this code:

<source lang="php">
<?php
require 'database.php';

$stmt = $mysqli->prepare("select first_name, last_name from employees order by last_name");
if(!$stmt){
printf("Query Prep Failed: %s\n", $mysqli->error);
exit;
}

$stmt->execute();

$result = $stmt->get_result();

echo "<ul>\n";
while($row = $result->fetch_assoc()){
printf("\t<li>%s %s</li>\n",
htmlspecialchars( $row["first_name"] ),
htmlspecialchars( $row["last_name"] )
);
}
echo "</ul>\n";

$stmt->close();
?>
</source>

Instead of binding the columns of the result set to variables, they are returned in an associative array. '''$result->fetch_assoc()''' returns a new array for each row in the result set, and when there are no more rows, it returns null, thereby exiting the loop.

==== Troubleshooting ====

If you are getting an error <code>call to undefined method mysqli_stmt::get_result()</code>, you probably installed the '''php(5)-mysql''' instead of the '''php(5)-mysqlnd''' package. To fix this, first uninstall the old package and then install the new one.

<source lang="bash">
$ sudo yum remove php-mysql
$ sudo yum install php-mysqlnd
</source>

== Combined Queries ==

You can also perform queries that both take input and produce output; you simply call both '''$stmt->bind_param()''' and '''$stmt->bind_result()'''. For example, the following selects all employees in the department specified in the GET variable '''dept''':

<source lang="php">
<?php
require 'database.php';

$dept = $_GET['dept'];

$stmt = $mysqli->prepare("select first_name, last_name from employees where department=?");
if(!$stmt){
printf("Query Prep Failed: %s\n", $mysqli->error);
exit;
}

$stmt->bind_param('s', $dept);

$stmt->execute();

$stmt->bind_result($first, $last);

echo "<ul>\n";
while($stmt->fetch()){
printf("\t<li>%s %s</li>\n",
htmlspecialchars($first),
htmlspecialchars($last)
);
}
echo "</ul>\n";

$stmt->close();

?>
</source>

[[Category:Module 3]]

SSH

2023-09-01T20:33:48Z

Zach: /* SSH Configuration */

Secure SHell, or '''SSH''', is the leading interface for connecting as an administrator from your local computer to something on the cloud. You will be using exclusively SSH to execute commands and upload files to your cloud instance. If you use a Git hosting service like BitBucket or Github, you will need SSH in order to save your changes there as well.

SSH access requires that the ''sshd'' daemon is running on the remote machine. By default, SSH is preinstalled on your EC2 instance. If you are not using an EC2 instance, simply install '''ssh''' from yum or apt on the remote machine.

== SSH Keys ==

SSH can work using password-based authentication, but it is more common nowadays, and also more secure, to use ''public and private keys'', also known as a ''key pair''.

Here's how it works. You have a private copy of your key, called your '''private key'''. Servers to which you want to connect have an analog to your private key called your '''public key'''. When you attempt to connect to a server using your private key, the server checks to see if any of its public keys "fits" your private key, and if it finds a match, it lets you in.

You can think of the ''private key'' as the "key" and the ''public key'' as the "lock".

== SSH Configuration ==

This section discusses how to set up your computer and your remote instance for SSH. You need to do this only once.

{{RequiredInstructions|content=

=== Creating an SSH Key Pair ===

You should create your own SSH Key Pair. You will keep the private key on your own computer, and you will upload the public key to all places to which you need to connect.

To generate a key pair, open a terminal ([[Workflow#Installing_a_Terminal_Emulator|on Windows, you'll need to have enabled WSL or installed Cygwin]] and run the following command:

<source lang="bash"># Run on Local Machine in Terminal or Cygwin
$ ssh-keygen</source>

'''Important:''' Don't name the file unless you are using a CEC machine. If you are using a CEC machine, name the file ''/cygdrive/h/.ssh/id_rsa''

For the purposes of this course, you do not need to enter a passphrase, although you would probably want one in real life.

Your keys should have been created in ~/.ssh. Check to make sure they exist using the ''ls'' command:

<source lang="bash">
$ ls ~/.ssh
id_rsa id_rsa.pub
</source>

''id_rsa'' is your private key, and ''id_rsa.pub'' is the corresponding public key. Let's look at the public signature of your key pair:

<source lang="bash">
$ cat ~/.ssh/id_rsa.pub
ssh-rsa AAAA3Bblahblahblahblah yourname@somedomain
</source>

We will need this later. Let's now add our key to the ''ssh-agent''.

==== SSH Agent: OS X ====

Run the following command to add your key to the SSH agent in Mac OS X.

<source lang="bash"># Non-Cygwin
$ ssh-add</source>

==== SSH Agent: WSL or Cygwin ====

In order to make the SSH agent run when you start your terminal, you need to run a couple more commands (these will add a line to your .bashrc file, which runs automatically on startup):

<source lang="bash">
# Cygwin only
$ echo "eval \`ssh-agent -s\`; ssh-add" >> ~/.bashrc
$ source ~/.bashrc # reload the Bash shell
</source>

If you get this scary-looking error after you run the second command…

<source lang="text">
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: UNPROTECTED PRIVATE KEY FILE! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
</source>

…then you need to do the following to fix the permissions:

<source lang="bash">
$ chgrp Users </path/to/id_rsa>
$ chmod 600 </path/to/id_rsa>
</source>

=== Amazon EC2 SSH Configuration ===

When you first created your EC2 instance, you downloaded a *.pem file. This is the ''private key'' for the default user ('''ec2-user''' on AMI). We need use this key to connect to your instance using the default user, at which point we will create a new custom user. We will give that new user the public key that you created in the previous step. You will then be able to log into your custom user using your personal key pair, so you won't need the default user again (although you should keep the *.pem file around just in case).

==== Logging In as the Default User ====

To connect to the default user on your Amazon EC2 instance, run this command in your terminal:

<source lang="bash">
# Run on Local Machine
$ ssh -i </path/to/default-privatekey.pem> ec2-user@<ec2-xx-xx-xx-xx.compute-1.amazonaws.com>
</source>

'''Important:''' As soon as you are able to successfully log in, commands you type into your prompt will be executed on the remote instance, ''not'' your local machine.

If you get this scary-looking warning:

<source lang="text">
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: UNPROTECTED PRIVATE KEY FILE! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
</source>

…then you need to correctly set the permissions on your *.pem file. You should know how to do this from the Linux guide, but in case you didn't catch that detail, run:

<source lang="bash">
# Run on Local Machine

# Cygwin Only:

$ chgrp Users </path/to/default-privatekey.pem>

# For Everyone:
$ chmod 600 </path/to/default-privatekey.pem>
</source>

'''Food for Thought:''' Why you wouldn't want other people viewing your private key file?

==== Creating a New User with SSH Permissions ====

First, you need to create your new user with sudo and SSH permissions. Again, the instructions for this are in the Linux guide, but to keep everything in one place, the commands are:

<source lang="bash">
# Run these on your instance, not on your local machine!

# Don't forget to customize `<My Full Name>` and `<usernameHere>`:
$ sudo useradd -r -m -c "<My Full Name>" <usernameHere>
$ sudo passwd <usernameHere>

# You need to add your username to the sudoers list. Refer to the Linux guide for details.
# Look around line 101.
$ sudo visudo

# Restart the SSH server:
$ sudo service ssh restart || sudo service sshd restart
</source>

Okay, cool: we have created our custom username on our remote instance. Now, we need to enable this account to accept our SSH key when authenticating. To do this, we need to add our public key to a file named '''authorized_keys''', which is stored in your remote username's SSH configuration directory.

To create the directory and edit the file, run on your instance:

<source lang="bash">
# Switch to our own user to ensure that permissions are set correctly:
$ su <username here>

# Change the working directory to our user's home directory:
$ cd

# Make the nonexistent SSH configuration directory and set its permissions:
$ mkdir .ssh
$ chmod 700 .ssh

# Create the `authorized_keys` file and set the permissions
$ touch .ssh/authorized_keys
$ chmod 600 .ssh/authorized_keys

# Edit the `authorized_keys` file
$ nano .ssh/authorized_keys
</source>

For more detail on editing files on the command line, see [[Linux#File Editors|the Linux guide]].

Inside of the `authorized_keys` file, paste in your public key. Your public key is the content of `~/.ssh/id_rsa.pub` '''on your local machine''' (it starts with "ssh-rsa"). There are a few ways to get it onto your clipboard:

# '''OS X Only:''' Open a new Terminal and run: <source lang="bash">cat ~/.ssh/id_rsa.pub | pbcopy</source> This immediately copies the contents of your public key to your clipboard.
# '''Windows Only:''' Navigate to id_rsa.pub manually, open it in your favorite text editor (like Notepad++), select all, and copy. The location of id_rsa.pub might be: ''C:\cygwin\home\username\.ssh\id_rsa.pub''
# '''*NIX Only:''' Open a new Terminal and copy the output of the following command: <source lang="bash">cat ~/.ssh/id_rsa.pub</source> Note that this won't work as advertised in Cygwin because Cygwin adds extra line breaks when you try to copy out of it.

When finished, save and close `authorized_keys`.

'''Food for Thought:''' How would you enable your remote user to authenticate over SSH using the same default key instead of the one you created using ''ssh-keygen''?

==== Logging In as your New User ====

Let's test whether we can log into our instance using our own username and our own key pair. Open up a new Terminal or Cygwin window '''on your local machine''' (not from ec2) and run:

<source lang="bash">
$ ssh <username here>@<ec2-xx-xx-xx-xx.amazonaws.com>
</source>

This is how you will be SSHing into your instances for the remainder of the semester.

}}

=== Troubleshooting ===

Almost everyone receives the following error at some point in their career: <code>Permission denied (publickey)</code>.

Here are some possible causes for the error.

==== The Authorized Keys File ====

Make sure that your remote ''authorized_keys'' file contains ''exactly'' your public signature from ''id_rsa.pub''.

===== Pasting into Vi =====

If you used vi to edit your file and you forgot to enter "insert" mode before pasting in your key, it is possible that you lost the first "s" in "ssh-rsa".

Check the contents of authorized_keys via <code>cat ~/.ssh/authorized_keys</code> on your server. If the output starts with "sh-rsa" instead of "ssh-rsa", edit the authorized_keys file and add the second '''s'''.

===== Unintentional Line Breaks =====

If you got your key via a <code>cat ~/.ssh/id_rsa.pub</code> and then copied it from the terminal, it's possible that you unintentionally introduced line breaks. Ensure that the key is represented by only one line of text.

==== Permissions ====

They need to be correct on both the client side and on the server side.

* Local ''.ssh'' = 700
* Local ''id_rsa'' = 600
* Remote ''.ssh'' = 700
* Remote ''authorized_keys'' = 600

If you are in Cygwin and the permissions don't seem to change, you might need to add the troubled file to a usergroup, like this:

<source lang="bash">
$ chgrp Users .ssh/id_rsa
</source>

==== The SSH Agent ====

You need to be sending the correct key along with your SSH Agent. Run the following command, and you should see some output:

<source lang="bash">
$ ssh-add -l
2048 ab:cd:ef:ab:cd:ef:ab:cd:ef:ab:cd:ef:ab:cd:ef:ab:cd /path/to/.ssh/id_rsa (RSA)
</source>

If you don't see any output, or if the output does not contain your ''id_rsa'', then you need to run:

<source lang="bash">
$ ssh-add </path/to/.ssh/id_rsa>
</source>

=== More SSH Server Configurations ===

Up to this point, we have been using all of the default SSH configurations. Like most things in Linux, SSH can be customized.

For the purposes of CSE 330, the additional configurations in this short section are optional; this serves only as a reference.

==== Disabling Root Access ====

It is almost always a good idea to disable root access over ssh. This could be done by editing ''/etc/ssh/sshd_config'' and setting

<source lang="text">PermitRootLogin no</source>

You will need to restart the SSH server for changes to take effect:

<source lang="bash">
$ sudo service sshd restart
</source>

==== Enabling Password-Based Authentication ====

Up to this point, we have been using key pairs for SSH. A public/private key pair is generally considered to be more secure, but it requires that you always have access to your private key file when you want to log into your remote machine.

By default, EC2 instances allow only public/private key pair authentication. You can enable password-based authentication by setting the PaswordAuthentication option in ''/etc/ssh/sshd_config'' to '''yes''':

<source lang="text">PasswordAuthentication yes</source>

There may be a line with this option that you can un-comment. For me, it is line 25. You will need to restart the server for changes to take effect.

If possible, however, you should restrict yourself to using private and public keys.

== Using SSH ==

Once you have completed the [[#SSH Configuration]] section above, all you need to do in order to SSH into your instance is to open a terminal and run:

<source lang="bash">
$ ssh <username here>@<ec2-xx-xx-xx-xx.compute-1.amazonaws.com>
</source>

=== PuTTY ===

'''This short section about PuTTY serves only as a reference.'''

If you have Cygwin available, you should use it to SSH into your instance. However, if you are using a different computer, you might not have Cygwin at your fingertips. In this case, a lighter-weight SSH client called '''PuTTY''' is available for Windows.

You can download PuTTY from: http://www.chiark.greenend.org.uk/~sgtatham/putty/

Amazon provides [http://docs.amazonwebservices.com/AWSEC2/latest/GettingStartedGuide/ConnectToInstanceLinux.html#connect-from-windows-machine a great tutorial on how to connect to a virtual machine from Windows].

PuTTY is fairly simple and straight forward with one caveat: Amazon's ''*.pem'' files are SSH private key files, and they need to be converted to PuTTY's own '''.ppk''' format. To do this, use the ''puttygen.exe'' utility available from the same page as PuTTY. Select "Import" under the conversions menu, load the amazon ''*.pem'' key file, and press the "Save Private Key" button.

Copy and paste works similarly to the X Window System in Unix. You use the left mouse button to select text in the PuTTY window. The act of selection automatically copies the text to the clipboard: there is no need to press Ctrl-Ins or Ctrl-C or anything else. In fact, pressing Ctrl-C will send a Ctrl-C character to the other end of your connection (just like it does the rest of the time), which may have unpleasant effects. The only thing you need to do, to copy text to the clipboard, is to select it.

To paste the clipboard contents into a PuTTY window, by default you click the right mouse button. If you have a three-button mouse and are used to X applications, you can configure pasting to be done by the middle button instead, but this is not the default because most Windows users don't have a middle button at all.

Here is a good PuTTY tutorial that you might find useful to get started: http://kb.mediatemple.net/questions/1595/Using+SSH+in+Putty+%28Windows%29

== SSHFS and SFTP ==

In addition to just SSH, your SSH server also supports '''SSHFS''', which enables you to mount your remote instance as a disk on your local computer, and '''SFTP''', which is a file transfer protocol that enables you to upload files directly to your instance.

=== SSHFS ===

SSHFS is a filesystem client which allows secure mounting of remote file systems. While there are other ways to mount remote file systems, SSHFS has the advantage of being able to mount a file system located on any host that has an SSH daemon running without any host side installation or configuration. This means that you can easily access and edit your files using all of your local applications including IDEs.

As you may have inferred from the name, the underlying implementation utilizes SSH File Transfer Protocol in combination with FUSE, a package now included in the kernel that allows unprivileged users to easily create their own file systems in userspace (see the wikipedia entry for more information [http://en.wikipedia.org/wiki/Filesystem_in_Userspace]).

To mount a share using password based authentication, the command is:

<source lang="bash">
$ sshfs <user>@<domain>:</path/to/remote/directory> </path/to/local/mountpoint>
</source>

For example, to mount the directory /home/joe/myfiles in the user ''joe'''s home directory for a machine with the domain schmoesfiles.org using SSHFS you would enter the command

<source lang="bash">
$ sshfs joe@www.schmoesfiles.org:myfiles
</source>

Note that if you are using public key authentication, the command to mount the remote filesystem may need to be slightly different:

<source lang="bash">
$ sshfs -o IdentityFile=</path/to/private/key> <user>@<domain>:</path/to/remote/directory> </path/to/local/mountpoint>
</source>

To unmount the filesystem, you can use the following command:

<source lang="bash">
$ fusermount -u </path/to/local/mountpoint>
</source>

=== SFTP ===

Any server running an SSH server is also compatible with '''SFTP''' or Secure File Transfer Protocol. (Compare to FTP, or File Transfer Protocol.) SFTP is a convenient way to edit files on your computer and then upload them to your server in just a few clicks.

[[Category:Module 2]]

HTML Forms

2023-08-28T17:15:14Z

Zach: /* HTTP Headers */

The contents of a web page often changes depending on how you requested that page. For example, when you perform a web search, the contents of the search list changes depending on your query. This is done by way of HTTP request variables (about which you will learn more in Module 2).

The most fundamental way to make a web page sent HTTP request variables to another page is by using HTML Forms. This article describes this process and how to implement it in your code.

== HTTP Request Variables ==

Visit your favorite search engine, and search for "QUERY". Now, look at your address bar. Can you find your query "QUERY" somewhere in the URL?

Your search engine probably used an HTTP Request Variable to send your desired query to the search page. Here is how it might have worked:

[[File:HTMLForm.png]]

You can see that we didn't just go to example.com/search; we also incorporated the search query into the '''query string''' of the URL, where query stream

=== Code Example ===

The syntax for making a form in HTML is relatively straightforward. The HTML for the above example could be:

<source lang="html4strict">
<form action="http://example.com/search" method="GET">
<label>Search Query: <input type="text" name="q" /></label>
<input type="submit" value="Search" />
</form>
</source>

The '''<label>''' tag above allows text to "label" an input field. When anything in the label is clicked, the input field will become focussed.

== HTTP Request Methods ==

In the above example, we used the GET method for transferring variables from our form to our destination page. There are two widely supported options for transferring the variables:

# '''GET''' transfers the variables in the URL.
# '''POST''' transfers the variables as an HTTP header.

What does "as an HTTP header" mean?

=== HTTP Headers ===

Whenever you request a page on the http:// protocol, your browser sends a ''request header''. Your server responds with a ''response header'' followed by the content of your page. You never see the request or response headers unless you have tools that let you see them.

To let you have an idea, here are the request and response headers when I visit http://www.w3.org/ in Firefox:

'''Request Headers:'''
<source lang="text">
GET / HTTP/1.1
Host: www.w3.org
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:19.0) Gecko/20100101 Firefox/19.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive
If-Modified-Since: Sat, 16 Mar 2013 09:16:37 GMT
If-None-Match: "8c8c-4d80736774740;89-3f26bd17a2f00"
Cache-Control: max-age=0
</source>

'''Response Headers:'''
<source lang="text">
HTTP/1.1 200 OK
Date: Sat, 16 Mar 2013 19:42:55 GMT
Server: Apache/2
Content-Location: Home.html
Vary: negotiate,accept
TCN: choice
Last-Modified: Sat, 16 Mar 2013 09:16:37 GMT
Etag: "8cc6-4d80736774740;89-3f26bd17a2f00"
Accept-Ranges: bytes
Content-Length: 36038
Cache-Control: max-age=600
Expires: Sat, 16 Mar 2013 19:52:55 GMT
P3P: policyref="http://www.w3.org/2001/05/P3P/p3p.xml"
Connection: close
Content-Type: text/html; charset=utf-8
</source>

You can see that in the request header, information is sent about what browser and operating system I'm using (my ''user agent''), what language I prefer (in my case, ''en-US''), and the like. If you have cookies for a site, they get transfered via a request header. In the response header, we can see that the request was successful (HTTP code 200), that I am receiving an HTML document with the UTF-8 charset (in ''Content-Type''), and other various information. All of these headers are standard; you can implement less-common headers as well.

=== When to use GET vs. POST ===

Here are some rules of thumb to help you decide whether to use GET or POST for a certain form.

* If you would like the user of your application to be able to ''bookmark the destination page'' for later use (e.g. search results), you need to use '''GET''', because the bookmark will contain the entire URL, including the request variables.
* If you are transmitting ''sensitive information'' like passwords, you should use '''POST''', because POST variables will not appear in a user's browsing history, but GET variables will.
* If the data you are transmitting is large (over about 200 bytes), you need to use '''POST''', since some browsers will truncate the URL string after 256 characters.
* If the destination page takes the information from the form and modifies something in a database, it is best practice to use '''POST'''.

=== Technicalities ===

Although it is safe to think about POST variables as being transmitted as a header, this is technically not the case. In reality, POST variables are sent in the request body. The server knows to read the POST variables from the request body because of a '''application/x-www-form-urlencoded''' request content type header.

Additionally, there has been a recent resurgence in so-called [[wikipedia:CRUD|CRUD]]; that is, additional HTTP request types beyond GET and POST: PUT, DELETE, OPTIONS, etc. CRUD request types are beyond the scope of this course, but it is worth the time reading up on their meaning if you find yourself working with a RESTful web framework like Ruby on Rails.

== Input Types ==

In the example above, we used just two input types: a '''text''' field and a '''submit''' button. In reality, there are over 20 different types. Some of the more common types include:

* '''hidden:''' allows for a variable to be passed as part of a form without requiring that the user input any data.
* '''password:''' allows for text to be hidden while typed.
* '''number:''' like a text box, but forces the content to be numerical.
** ''Note:'' You can set the '''step''', '''min''', and '''max''' attributes on your '''number''' input to change the precision of the number accepted. For example, the following input field would accept only integers between 0 and 100 inclusive:
**: <source lang="html4strict"><input type="number" name="percentage" step="1" min="0" max="100" /></source>
* '''checkbox:''' a yes-or-no checkbox.
* '''radio:''' a multiple-choice question. Use multiple input fields, all with the same ''name'' but different ''value''.
* '''file:''' allows for file uploads. (We will deal with this more in module 2.)

For a complete list, see [https://www.w3.org/TR/html5/forms.html#the-input-element section 4.10.7 of the HTML5 spec].

[[Category:Module 1]]

M1 Introduction to Git

2023-08-15T20:41:45Z

Zach: /* Git */

'''NOTE: This is an extremely simplified guide to Git meant to kickstart your work in this course. We ''strongly'' recommend you read the [[M1 Git Extended|Git Extended]] reading.'''

== Git ==

=== Installation ===
Git may already be installed on your device. You can check this by running
<source lang="bash">git version</source>

If it's not installed...

'''On your AWS Instance'''
<source lang="bash">
# install git using the "yum" linux package manager
$ sudo yum install git
</source>

'''After installation, make sure you complete this final step'''
<source lang="bash">
# Set the name and email for your git usage.
# This does not have to match your GitHub account

# You can think of this as signing your name on your changes

$ git config --global user.name "<Your Name>"
$ git config --global user.email "<you@example.com>"
</source>

=== Why Git? ===

Git is a very powerful tool that's critical to modern software development. It's significance is rooted in three core features:

==== Distributed Version Control ====
Think of Git as a cookbook shared among several chefs. Each chef has a complete copy of the recipes, so they can all experiment and make changes without waiting for one another. If one chef spills sauce on their copy, the others are unaffected. Likewise, each chef can peek into each other's copies to see new recipes or changes.

In traditional centralized version control, there is a single "master" copy of the code. In contrast, Git's distributed nature means that every developer has a full-fledged copy of not only the entire codebase but also all the branches within it, including their respective histories. This structure allows more parallel development, supports experimentation with different parts of the code in isolation, and reduces reliance on a central server, thereby minimizing potential points of failure.

==== History and Tracking ====
Imagine your code as a time-traveling book where you can flip back to see what was written at any point in time. You can even find out who wrote each sentence and why.

Git keeps a detailed log of every change made to the code, including who made it and why (via commit messages). This allows for greater accountability and ease of tracking changes over time. If a bug is discovered, developers can quickly identify when it was introduced and by whom, facilitating a quicker resolution.

==== Branching and Merging ====
You can think of git branches as individual workspaces within a repository. Each branch allows for independent development, experimentation, or bug-fixing, and acts like a parallel universe to the main codebase. Changes made within a branch do not affect other branches, giving developers the freedom to explore and innovate without risk to the stable versions of the project. When a feature is completed or a bug is fixed, the branch can be merged back into the main codebase, seamlessly integrating the updates.

===== Conflict Resolution =====
''' Without Merge Conflicts '''

Imagine two developers working on two new features, each in their own branch. If both developers are working on different parts of the codebase, or different files altogether, their changes can typically be merged back into the main codebase without any conflicts. The modifications are independent of each other and do not overlap, allowing for a seamless merge.

'''With Merge Conflicts'''

Now consider a scenario where both developers are working on the same part of the code or the same file within their respective branches. When they try to merge their changes back into the main codebase, there may be conflicts where their changes overlap. Git will not know which changes to keep and which to discard, leading to a merge conflict.

In this situation, manual intervention is needed to resolve the conflicts. The developers must review the conflicting changes, decide which to keep or how to combine them, and then commit the resolved code. Command line tools and graphic interfaces (likely built into your IDE) can assist in this process, but understanding the context of the changes and clear communication between developers is crucial for a successful resolution.

M1 Introduction to Git

2023-08-15T20:24:30Z

Zach: /* Conflict Resolution */

'''NOTE: This is an extremely simplified guide to Git meant to kickstart your work in this course. We ''strongly'' recommend you read the [[M1 Git Extended|Git Extended]] reading.'''

== Git ==

'''Git''' is a ''distributed version control system.'' It enables you to:
# Keep track of revisions to your files
# Push and pull your revisions to other computers or servers
# Collaborate with other people on the same project

=== Why Git? ===

Git is a very powerful tool that's critical to modern software development. It's significance is rooted in three core features:

==== Distributed Version Control ====
Think of Git as a cookbook shared among several chefs. Each chef has a complete copy of the recipes, so they can all experiment and make changes without waiting for one another. If one chef spills sauce on their copy, the others are unaffected. Likewise, each chef can peek into each other's copies to see new recipes or changes.

In traditional centralized version control, there is a single "master" copy of the code. In contrast, Git's distributed nature means that every developer has a full-fledged copy of not only the entire codebase but also all the branches within it, including their respective histories. This structure allows more parallel development, supports experimentation with different parts of the code in isolation, and reduces reliance on a central server, thereby minimizing potential points of failure.

==== History and Tracking ====
Imagine your code as a time-traveling book where you can flip back to see what was written at any point in time. You can even find out who wrote each sentence and why.

Git keeps a detailed log of every change made to the code, including who made it and why (via commit messages). This allows for greater accountability and ease of tracking changes over time. If a bug is discovered, developers can quickly identify when it was introduced and by whom, facilitating a quicker resolution.

==== Branching and Merging ====
You can think of git branches as individual workspaces within a repository. Each branch allows for independent development, experimentation, or bug-fixing, and acts like a parallel universe to the main codebase. Changes made within a branch do not affect other branches, giving developers the freedom to explore and innovate without risk to the stable versions of the project. When a feature is completed or a bug is fixed, the branch can be merged back into the main codebase, seamlessly integrating the updates.

===== Conflict Resolution =====
''' Without Merge Conflicts '''

Imagine two developers working on two new features, each in their own branch. If both developers are working on different parts of the codebase, or different files altogether, their changes can typically be merged back into the main codebase without any conflicts. The modifications are independent of each other and do not overlap, allowing for a seamless merge.

'''With Merge Conflicts'''

Now consider a scenario where both developers are working on the same part of the code or the same file within their respective branches. When they try to merge their changes back into the main codebase, there may be conflicts where their changes overlap. Git will not know which changes to keep and which to discard, leading to a merge conflict.

In this situation, manual intervention is needed to resolve the conflicts. The developers must review the conflicting changes, decide which to keep or how to combine them, and then commit the resolved code. Command line tools and graphic interfaces (likely built into your IDE) can assist in this process, but understanding the context of the changes and clear communication between developers is crucial for a successful resolution.

M1 Introduction to Git

2023-08-15T20:23:50Z

Zach: /* = Distributed Version Control */

'''NOTE: This is an extremely simplified guide to Git meant to kickstart your work in this course. We ''strongly'' recommend you read the [[M1 Git Extended|Git Extended]] reading.'''

== Git ==

'''Git''' is a ''distributed version control system.'' It enables you to:
# Keep track of revisions to your files
# Push and pull your revisions to other computers or servers
# Collaborate with other people on the same project

=== Why Git? ===

Git is a very powerful tool that's critical to modern software development. It's significance is rooted in three core features:

==== Distributed Version Control ====
Think of Git as a cookbook shared among several chefs. Each chef has a complete copy of the recipes, so they can all experiment and make changes without waiting for one another. If one chef spills sauce on their copy, the others are unaffected. Likewise, each chef can peek into each other's copies to see new recipes or changes.

In traditional centralized version control, there is a single "master" copy of the code. In contrast, Git's distributed nature means that every developer has a full-fledged copy of not only the entire codebase but also all the branches within it, including their respective histories. This structure allows more parallel development, supports experimentation with different parts of the code in isolation, and reduces reliance on a central server, thereby minimizing potential points of failure.

==== History and Tracking ====
Imagine your code as a time-traveling book where you can flip back to see what was written at any point in time. You can even find out who wrote each sentence and why.

Git keeps a detailed log of every change made to the code, including who made it and why (via commit messages). This allows for greater accountability and ease of tracking changes over time. If a bug is discovered, developers can quickly identify when it was introduced and by whom, facilitating a quicker resolution.

==== Branching and Merging ====
You can think of git branches as individual workspaces within a repository. Each branch allows for independent development, experimentation, or bug-fixing, and acts like a parallel universe to the main codebase. Changes made within a branch do not affect other branches, giving developers the freedom to explore and innovate without risk to the stable versions of the project. When a feature is completed or a bug is fixed, the branch can be merged back into the main codebase, seamlessly integrating the updates.

===== Conflict Resolution =====
''' Without Merge Conflicts '''
Imagine two developers working on two new features, each in their own branch. If both developers are working on different parts of the codebase, or different files altogether, their changes can typically be merged back into the main codebase without any conflicts. The modifications are independent of each other and do not overlap, allowing for a seamless merge.

'''With Merge Conflicts'''
Now consider a scenario where both developers are working on the same part of the code or the same file within their respective branches. When they try to merge their changes back into the main codebase, there may be conflicts where their changes overlap. Git will not know which changes to keep and which to discard, leading to a merge conflict.

In this situation, manual intervention is needed to resolve the conflicts. The developers must review the conflicting changes, decide which to keep or how to combine them, and then commit the resolved code. Command line tools and graphic interfaces (likely built into your IDE) can assist in this process, but understanding the context of the changes and clear communication between developers is crucial for a successful resolution.

M1 Introduction to Git

2023-08-15T20:22:54Z

Zach: /* Git */

'''NOTE: This is an extremely simplified guide to Git meant to kickstart your work in this course. We ''strongly'' recommend you read the [[M1 Git Extended|Git Extended]] reading.'''

== Git ==

'''Git''' is a ''distributed version control system.'' It enables you to:
# Keep track of revisions to your files
# Push and pull your revisions to other computers or servers
# Collaborate with other people on the same project

=== Why Git? ===

Git is a very powerful tool that's critical to modern software development. It's significance is rooted in three core features:

==== Distributed Version Control ===
Think of Git as a cookbook shared among several chefs. Each chef has a complete copy of the recipes, so they can all experiment and make changes without waiting for one another. If one chef spills sauce on their copy, the others are unaffected. Likewise, each chef can peek into each other's copies to see new recipes or changes.

In traditional centralized version control, there is a single "master" copy of the code. In contrast, Git's distributed nature means that every developer has a full-fledged copy of not only the entire codebase but also all the branches within it, including their respective histories. This structure allows more parallel development, supports experimentation with different parts of the code in isolation, and reduces reliance on a central server, thereby minimizing potential points of failure.

==== History and Tracking ====
Imagine your code as a time-traveling book where you can flip back to see what was written at any point in time. You can even find out who wrote each sentence and why.

Git keeps a detailed log of every change made to the code, including who made it and why (via commit messages). This allows for greater accountability and ease of tracking changes over time. If a bug is discovered, developers can quickly identify when it was introduced and by whom, facilitating a quicker resolution.

==== Branching and Merging ====
You can think of git branches as individual workspaces within a repository. Each branch allows for independent development, experimentation, or bug-fixing, and acts like a parallel universe to the main codebase. Changes made within a branch do not affect other branches, giving developers the freedom to explore and innovate without risk to the stable versions of the project. When a feature is completed or a bug is fixed, the branch can be merged back into the main codebase, seamlessly integrating the updates.

===== Conflict Resolution =====
''' Without Merge Conflicts '''
Imagine two developers working on two new features, each in their own branch. If both developers are working on different parts of the codebase, or different files altogether, their changes can typically be merged back into the main codebase without any conflicts. The modifications are independent of each other and do not overlap, allowing for a seamless merge.

'''With Merge Conflicts'''
Now consider a scenario where both developers are working on the same part of the code or the same file within their respective branches. When they try to merge their changes back into the main codebase, there may be conflicts where their changes overlap. Git will not know which changes to keep and which to discard, leading to a merge conflict.

In this situation, manual intervention is needed to resolve the conflicts. The developers must review the conflicting changes, decide which to keep or how to combine them, and then commit the resolved code. Command line tools and graphic interfaces (likely built into your IDE) can assist in this process, but understanding the context of the changes and clear communication between developers is crucial for a successful resolution.

M1 Markdown

2023-08-08T01:33:49Z

Zach: Created page with "= Markdown = Markdown is a lightweight markup language that allows you to create formatted text using a plain text editor. It uses simple symbols and characters to define dif..."

= Markdown =

Markdown is a lightweight markup language that allows you to create formatted text using a plain text editor. It uses simple symbols and characters to define different elements of formatting, such as headers, lists, links, and emphasis. Markdown files have the extension <source enclose="none">.md</source>

== Why Markdown? ==

* Simplicity: Markdown's syntax is easy to learn and write, even for those unfamiliar with HTML.
* Portability: Markdown files are plain text, so they can be opened and edited with any text editor.
* Versatility: It's widely used in various platforms like GitHub, where it's commonly used for README files and comments.
* Versatility: Even in raw form, Markdown is designed to be readable as-is, without looking like it's been marked up with tags or formatting instructions.

Writing in markdown allows for richly formatted text that's simple, efficient, and easily converted to HTML or other formats.

== Syntax ==
Instead of writing a markdown guide here, there are a bunch of great resources online:

- [https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax Github Markdown Guide]
- [https://www.markdownguide.org/ Markdown Guide]

M1 Introduction to Git

2023-08-05T02:16:42Z

Zach: /* Git */

'''NOTE: This is an extremely simplified guide to Git meant to kickstart your work in this course. We ''strongly'' recommend you read the [[M1 Git Extended|Git Extended]] reading.'''

== Git ==

'''Git''' is a ''distributed version control system.'' It enables you to:
# Keep track of revisions to your files
# Push and pull your revisions to other computers or servers
# Collaborate with other people on the same project

M1 Introduction to Git

2023-08-05T02:14:54Z

Zach: Created page with "'''NOTE: This is an extremely simplified guide to Git meant to kickstart your work in this course. We ''strongly'' recommend you read the Git Extended read..."

'''NOTE: This is an extremely simplified guide to Git meant to kickstart your work in this course. We ''strongly'' recommend you read the [[M1 Git Extended|Git Extended]] reading.'''

== Git ==

'''Git''' is a ''distributed version control system.'' It enables you to:
# Keep track of revisions to your files
## Push and pull your revisions to other computers or servers
### Collaborate with other people on the same project

M1 Validation

2023-08-05T01:51:01Z

Zach: /* Validation */

== Validation ==
{{XKCD
|id=1144
|name=tags
}}

HTML and to a lesser extent CSS are forgiving when it comes to syntactical errors in your code. However, it is not good practice to ignore your syntax errors for a variety of reasons:

* '''Faster and More Reliable Rendering.''' Don't leave it up to your client's browser to decide what you meant when you made your syntax error.
* '''Interoperability with Compliant Browsers.''' Although a web page may ''look'' right in your ''testing'' browser, you need to remember that there are ''lots'' of different user agents: other browsers, mobile devices, and screen readers, just to name a few.
* '''Accessibility.''' Valid HTML code enables you to be confident that your page will be accessible to people with disabilities, such as the blind.
* '''Search Engine Optimization.''' Web sites with valid HTML syntax rank higher in search engines like Google.
* '''Future Proofing.''' The W3C specification will not be made backwards-incompatible, so writing valid HTML syntax ensures that your page will work well into the future.
* '''More Professional.''' No seasoned web developer will publish a page with accidental syntax errors.

The industry standard tool for HTML and CSS syntax checking is the '''[https://validator.w3.org/ W3C Validator]'''. '''You should ALWAYS validate your code before completing your project!''' The Validator gives very helpful feedback to help you find and fix syntax errors.

For more information on why it is important to validate, see [https://validator.w3.org/docs/why.html the W3C's recommendation].

=== But why don't… ===

As you play around with the validator, you may have discovered that household name sites like [https://www.google.com/ Google] and [https://www.yahoo.com/ Yahoo] do not pass the validator. This is a deliberate act on the part of these web sites because of the following reasons:

* '''Download Speed.''' Sometimes, but not always, valid code adds bytes to the file size. Sites that serve tens of thousands of requests per second are highly concerned with the increased file size.
* '''Interoperability with Non-Compliant Browsers.''' ''Very'' old browsers like Netscape Navigator and Internet Explorer 5 do not respond the way that modern browsers do to the same markup.

However, do not take these as reasons to not validate your own code. The reasons listed in the previous section almost always outweigh the counterpoints in this section.

=== How to Validate Your Site ===
For HTML, use [https://validator.w3.org/ the HTML validator].
For CSS, use [https://jigsaw.w3.org/css-validator/ the CSS validator].

For the purposes of this course, a page is considered to pass the validator if the validator returns no errors or warnings. You will not be penalized for any "Info" type messages, but it's suggested you do your best to resolve them.

M1 Validation

2023-08-05T01:50:15Z

Zach: /* Validation */

M1 Validation

2023-08-05T01:45:31Z

Zach: Created page with "== Validation == {{XKCD |id=1144 |name=tags }} HTML and to a lesser extent CSS are forgiving when it comes to syntactical errors in your code. However, it is not good practi..."

Rewrite 2023

2023-08-05T01:39:26Z

Zach: /* Directory */

== 2023 Wiki reorg ==
I am working on a rewrite/reorg of the current wiki. In its current state, the wiki has an incredible amount of information, but I've heard from many students that it is difficult to parse and sometimes confusing as to which data is relevant. This effort serves to address some of these issues.

In its current state, students often find it difficult to separate the assignment-critical info from supplementary content. To address this, the rewrite seperates structures each assignment like this:

* Purpose (of module)
* Assignemnt
** Individual
** Group

* Reading
** Required
** Optional

=== Directory ===
* [[M1 Index|M1]]
** [[M1 Purpose|Purpose]]
** [[M1 Individual|Individual Assignment]]
** [[M1 Reading|Reading]]
*** Required
**** [[M1 Introduction to HTML|Introduction to HTML]]
**** [[M1 Introduction to CSS|Introduction to CSS]]
**** [[M1 Introduction to Git|Introduction to Git]]
**** [[M1 Validation|Validation]]
*** Supplemental
**** [[M1 Markdown|Markdown]]
**** [[M1 Accessibility|Accessibility]]
**** [[M1 HTML Extended|HTML Extended]]
**** [[M1 CSS Extended|CSS Extended]]
**** [[M1 Git Extended|Git Extended]]

<blockquote> ''More to be added soon. The other modules will follow the same basic structure.''</blockquote>

Rewrite 2023

2023-08-05T01:39:03Z

Zach: /* Directory */

== 2023 Wiki reorg ==
I am working on a rewrite/reorg of the current wiki. In its current state, the wiki has an incredible amount of information, but I've heard from many students that it is difficult to parse and sometimes confusing as to which data is relevant. This effort serves to address some of these issues.

In its current state, students often find it difficult to separate the assignment-critical info from supplementary content. To address this, the rewrite seperates structures each assignment like this:

* Purpose (of module)
* Assignemnt
** Individual
** Group

* Reading
** Required
** Optional

=== Directory ===
* [[M1 Index|M1]]
** [[M1 Purpose|Purpose]]
** [[M1 Individual|Individual Assignment]]
** [[M1 Reading|Reading]]
*** Required
**** [[M1 Introduction to HTML|Introduction to HTML]]
**** [[M1 Introduction to CSS|Introduction to CSS]]
**** [[M1 Introduction to Git|Introduction to Git]]
**** [[M1 Validation|Validation]]

*** Supplemental
**** [[M1 Markdown|Markdown]]
**** [[M1 Accessibility|Accessibility]]
**** [[M1 HTML Extended|HTML Extended]]
**** [[M1 CSS Extended|CSS Extended]]
**** [[M1 Git Extended|Git Extended]]

<blockquote> ''More to be added soon. The other modules will follow the same basic structure.''</blockquote>

M1 Introduction to CSS

2023-08-05T01:38:20Z

Zach: /* CSS (Cascading Style Sheets) */

== CSS (Cascading Style Sheets) ==

Notice that we haven't said a word about how things look yet. This is because that is what CSS is for!

CSS stands for ''Cascading Style Sheets'', and it defines the ''appearance'' of your page. CSS is composed of ''rules'', and each rule consists of a ''selector'' and one or more ''declarations''. Let's see an example:

<source lang="css">
header {
color: blue;
font-family: Verdana, sans-serif;
}
</source>

The above CSS rule does three things:

* It ''selects'' all <code><nowiki><header></nowiki></code> elements
* It ''declares'' the ''property'' '''color''' (i.e. the text color) of the selected elements to be blue
* It ''declares'' the ''property'' '''font-family''' (i.e. the font "face") of the selected elements to be Verdana, or alternatively any sans-serif font if Verdana is not available

All CSS rules follow this general format. The following sections will discuss more detail on CSS selectors and declarations.

=== Selectors ===

{{DoctypeTV
|youtube_id=W9H95NcbTSw
|time=5m27s
|desc=CSS Selectors, Part 1
}}

In the previous section's example, the most basic CSS selector was used: selecting by the ''tag name''. To select by the ''class name'', use a period, like so:

<source lang="css">
.info {
font-weight: bold;
}
</source>

{{DoctypeTV
|youtube_id=0a4r4bm1nQA
|time=1m29s
|desc=CSS Selectors, Part 2
}}

A third common way to select an element is by its ID. To select by ID, use a number sign, like so:

<source lang="css">
#medal_table {
border-collapse: collapse;
}
</source>

Selectors can be combined. For example, to select all hyperlinks in any paragraph having the class "alert", you could do:

<source lang="css">
p.alert a {
font-style: italic;
}
</source>

The <code>p.alert</code> selects all <code><nowiki><p></nowiki></code> tags with the class name "alert". The space tells CSS to look for all descendants of such paragraphs. Finally, the <code>a</code> selects all <code><nowiki><a></nowiki></code> elements in the resulting set.

It is also very common to use a right-angle bracket to select a ''child'' rather than a ''descendant'', the distinction being that ''descendants'' could be nested at any depth: child, grandchild, and so on, while a ''child'' must be at the next level down in the DOM. For example:

<source lang="css">
ul.fruits > li {
text-decoration: underline;
}
</source>

Some common CSS selectors are listed below. For a complete list of CSS selectors, see [https://www.w3.org/TR/CSS/#selectors section 4.2 of the CSS Specification].

* '''foo''' selects an element with tag name ''foo''
* '''.foo''' selects an element with the class name ''foo''
* '''#foo''' selects an element with the ID ''foo''
* '''[foo="bar"]''' selects an element with the attribute ''foo'' whose value is ''bar''
* '''foo bar''' selects an element matching ''bar'' that is a descendant of an element matching ''foo''
* '''foo > bar''' selects an element matching ''car'' that is a child of an element matching ''foo''
* '''foo + bar''' selects an element matching ''bar'' that is the next sibling of an element matching ''foo''

==== Pseudo-Classes ====

Elements can also be selected based on their current state using ''pseudo-classes''. Pseudo-classes always begin with a colon. For example, this is how you could change the color of all visited links on your web page:

<source lang="css">
a:visited {
color: #99FF99; /* this example uses a hexadecimal RGB definition for the color, which allows you to
mix over 16 million colors */
}
</source>

Some common pseudo-classes are listed below. For a complete list, see [https://www.w3.org/TR/CSS/#selectors section 4.2 of the CSS Specification].

* '''foo:first-child''' selects an element matching ''foo'' that is the first child of its parent (i.e., any ''foo'' that doesn't have older siblings)
* '''foo:last-child''' selects an element matching ''foo'' that is the last child of its parent
* '''foo:empty''' selects any element matching ''foo'' that has no children
* Pseudo-Classes for Form Elements:
** '''input:checked''' selects all checkboxes and radio buttons that are checked
* Pseudo-Classes for Hyperlink Elements:
** '''a:link''' selects all hyperlinks that have not been visited yet
** '''a:visited''' selects all hyperlinks that have been visited
* Pseudo-Classes for Interactive Elements (like buttons, hyperlinks, etc):
** '''foo:active''' selects elements matching ''foo'' that are being pressed on
** '''foo:hover''' selects elements matching ''foo'' that are being hovered over
** '''foo:focus''' selects elements matching ''foo'' that are selected (no pun intended)

=== Properties ===

Selectors are used to select elements that need to have styles applied to them, and properties are used to actually define those styles. Throughout this section, you've seen properties including '''color''', '''font-family''', '''font-style''', and '''text-decoration'''. There are dozens more properties you can define, some of which are listed below.

For a complete list of CSS properties, see [https://www.w3.org/TR/CSS/#properties section 4.1 of the CSS Specification].

* '''background-image''' sets the background image of an element. Usage: <code>background-image: url(path/to/image.png);</code>
* '''background-color''' sets the background color of an element (note: the image takes precedence over the color; the color will be displayed if the image can't be found)
* For Tables:
** '''border-collapse''' removes the spacing between table cells. Usage: <code>border-collapse: collapse;</code>
** Note: You can select td elements that are grandchildren of a table element, and then specify border properties on the td's, like this: <code>#myTable > tr > td { border: 1px solid black; }</code>
* '''border''' defines the border around an element. Usage: <code>border: 1px dashed red;</code>
* '''border-radius''' lets you have rounded corners on your elements
* '''float''' makes the text of a document flow around the element. Usage: <code>float: left;</code> (or <code>right</code>)
** '''clear''' makes an element stop wrapping around floats and instead move down to where the floats end. Usage: <code>clear: both;</code>
** Note: If you specify multiple object in a for as floating, they will appear next to each other horizontally.
* '''cursor''' lets you change the type of cursor that appears then your element is in focus. Example usage: <code>cursor: pointer;</code>
* '''font''' lets you specify the font-size, line-height, and font-family in one fell swoop. Usage: <code>font: 14px/18px Verdana, Arial, sans-serif;</code>
* '''list-style-type''' specifies the type of number or bullet used on <code><nowiki><ol></nowiki></code>'s and <code><nowiki><ul></nowiki></code>'s
* '''text-align''' lets you justify text on the left, center, or right
* '''display''' lets us specify an element as inline or block (see below), and it also enables us to "turn off" an element from being displayed on a page. Example usage: <code>display: none;</code>

==== Units ====

Some CSS properties require that you specify numerical values for things (font-size, line-height, border-width, and so on). CSS supports a number of units, the most common of which include:

* '''px''', which defines a distance in pixels (relative to screen resolution).
* '''em''', which defines a distance relative to font size. (Note: an ''em'' is the width of the capital letter M. No, I'm not kidding.)
* '''rem''', which defines a distance relative to the font size of the root element. Compare to '''em''', which defines a distance relative to the font size of the current element (which might have a different font size than the document root).
* '''pt''', which is a traditional print unit equal to 1/72 of an inch (should appear the same size on all screens regardless of resolution).
* '''%''', which defines a distance relative to that property of its parent element. For the most part, you'll only use % when working with the Box Model (see below).

=== The Box Model ===

For the most part, all elements fall into one of two categories:

* Inline (Examples: a, span, strong)
* Block (Examples: p, div, section)

The properties we've seen thus far apply mostly to inline elements. As it turns out, CSS has an additional set of properties that describe block elements. These properties compose the ''box model''.

While inline elements flow with the text, block elements stand on their own. You can define the space between the block element and the next closest element on the page using ''margin'', the block element's border using ''border'', and the space between the block element's border and its content using ''padding'':

[[File:Box.png]]

For example, you could make a padded box that is centered on the page using this CSS:

<source lang="css">
div.box {
width: 500px;
margin: 0 auto; /* when two "arguments" are passed to margin or padding, the first defines top
and bottom margin/padding and the second defines left and right margin/padding */
padding: 25px; /* when only one "argument" is passed to margin or padding, it defines the
margin/padding on all four sides */
border: 1px solid black; /* the 1px defines the border-width; the solid defines the border-style;
the black defines the border-color */
}
</source>

Here is what the resulting box looks like:

[[File:CSSBoxExample.png]]

Note that the ''width'' defines only the ''content width''. To compute the actual "width" of the element, you need to add the left and right padding border, and margin.

This is only scraping the surface of the CSS box model. All advanced web sites utilize the box model to its full extent to produce the complex layouts you have come to expect on the internet. For more information on the CSS box model, read [https://www.w3.org/TR/css3-box/ the W3C specification].

=== Typography ===

Traditionally, Web developers were limited to a small subset of fonts that were commonly installed on all computers:

* Andale Mono
* Arial
* Arial Black
* Comic Sans MS
* Courier New
* Georgia
* Impact
* Times New Roman
* Trebuchet MS
* Verdana
* Webdings

That was it. Incredibly limiting.

However, fonts other than these are used widely on the Web. How? CSS has evolved to the point where it now allows you to upload a font to your server, and use that font on your site, or even include CSS from another server that adds the fonts you want. One excellent resource for a wide variety of
fast, appropriately-licensed fonts is [https://fonts.google.com/ Google Fonts].

== Using HTML and CSS in your Web Page ==

Recall that HTML defines the ''content'' of your page, while CSS defines the ''appearance''.

Most of the HTML you write will be in the <code><body></code> tag, because, naturally, this is where the ''content'' of your page should reside. CSS, on the other hand, is supplemental to the content, and so it should be included in the <code><head></code> tag, as shown in the following section.

=== How to Include CSS ===

There are three options for using CSS in your web page:

# Loading external stylesheets
# Embedding stylesheets in the document
# Inline styles using the style attribute

The first option is almost always preferred in production applications for the following reasons:

* Browsers can cache external stylesheets, so once they've loaded the stylesheet once, they won't have to load it again, making future pages load faster and reducing requests to your server
* External stylesheets are more modular; that is, they can be included in multiple pages on your web site without the need for copying and pasting embedded stylesheets

To include external stylesheets in your document, place the following empty HTML tag in your <code><head></code>:

<source lang="html4strict">
<link rel="stylesheet" type="text/css" href="path/to/stylesheet.css" />
</source>

It may be convenient during development to use embedded stylesheets, because then you don't have to click back and forth between files in your IDE and clear your browser cache. You may place embedded stylesheets in a <code><style></code> tag, also in your <code><head></code>:

<source lang="html4strict">
<style type="text/css">
styles {
go: here;
}
</style>
</source>

For a variety of reasons, inline styles should almost always be avoided. Instead of using inline styles, give your element a CSS class, and assign styles to that class in your stylesheet:

<source lang="css">
p.notice {
color: blue;
}
</source>
<source lang="html4strict">
<p class="notice">Well done using classes instead of inline styles!</p>
</source>

Just so you know it when you see it, the following example shows the usage of inline styles, which are declared using the '''style''' attribute.

<source lang="html4strict">
<p style="color:red;">You'd better think twice about using inline styles.</p>
</source>

=== Quick and Easy Page Layout ===

I like to use the following code when I just want a simple, aesthetically pleasing web page:

<source lang="html4strict">
<!DOCTYPE html>
<head>
<meta charset="utf-8"/>
<title>My Web Page</title>
<style type="text/css">
body{
width: 760px; /* how wide to make your web page */
background-color: teal; /* what color to make the background */
margin: 0 auto;
padding: 0;
font:12px/16px Verdana, sans-serif; /* default font */
}
div#main{
background-color: #FFF;
margin: 0;
padding: 10px;
}
</style>
</head>
<body><div id="main">



</div></body>
</html>
</source>

Put your document where the <code><nowiki></nowiki></code> is. Your page will go instantly from dull-looking to presentable. ;-)

=== The HTML5 Shiv ===

Some browsers (*cough cough* IE < version 9) do strange things when you apply CSS styles to the newest HTML elements that were introduced with HTML5. Fortunately, there is a painless solution. Simply include the following code in your '''<head>''' tag:

<source lang="html4strict">

</source>

This so-called '''HTML5 Shiv''' will make older versions of Internet Explorer render your pages more closely to how modern browsers do.

M1 Introduction to CSS

2023-08-05T01:35:57Z

Zach: Created page with "== CSS (Cascading Style Sheets) == Notice that we haven't said a word about how things look yet. This is because that is what CSS is for! CSS stands for ''Cascading Style..."

== CSS (Cascading Style Sheets) ==

Notice that we haven't said a word about how things look yet. This is because that is what CSS is for!

CSS stands for ''Cascading Style Sheets'', and it defines the ''appearance'' of your page. CSS is composed of ''rules'', and each rule consists of a ''selector'' and one or more ''declarations''. Let's see an example:

<source lang="css">
header {
color: blue;
font-family: Verdana, sans-serif;
}
</source>

The above CSS rule does three things:

* It ''selects'' all <code><nowiki><header></nowiki></code> elements
* It ''declares'' the ''property'' '''color''' (i.e. the text color) of the selected elements to be blue
* It ''declares'' the ''property'' '''font-family''' (i.e. the font "face") of the selected elements to be Verdana, or alternatively any sans-serif font if Verdana is not available

All CSS rules follow this general format. The following sections will discuss more detail on CSS selectors and declarations.

=== Selectors ===

{{DoctypeTV
|youtube_id=W9H95NcbTSw
|time=5m27s
|desc=CSS Selectors, Part 1
}}

In the previous section's example, the most basic CSS selector was used: selecting by the ''tag name''. To select by the ''class name'', use a period, like so:

<source lang="css">
.info {
font-weight: bold;
}
</source>

{{DoctypeTV
|youtube_id=0a4r4bm1nQA
|time=1m29s
|desc=CSS Selectors, Part 2
}}

A third common way to select an element is by its ID. To select by ID, use a number sign, like so:

<source lang="css">
#medal_table {
border-collapse: collapse;
}
</source>

Selectors can be combined. For example, to select all hyperlinks in any paragraph having the class "alert", you could do:

<source lang="css">
p.alert a {
font-style: italic;
}
</source>

The <code>p.alert</code> selects all <code><nowiki><p></nowiki></code> tags with the class name "alert". The space tells CSS to look for all descendants of such paragraphs. Finally, the <code>a</code> selects all <code><nowiki><a></nowiki></code> elements in the resulting set.

It is also very common to use a right-angle bracket to select a ''child'' rather than a ''descendant'', the distinction being that ''descendants'' could be nested at any depth: child, grandchild, and so on, while a ''child'' must be at the next level down in the DOM. For example:

<source lang="css">
ul.fruits > li {
text-decoration: underline;
}
</source>

Some common CSS selectors are listed below. For a complete list of CSS selectors, see [https://www.w3.org/TR/CSS/#selectors section 4.2 of the CSS Specification].

* '''foo''' selects an element with tag name ''foo''
* '''.foo''' selects an element with the class name ''foo''
* '''#foo''' selects an element with the ID ''foo''
* '''[foo="bar"]''' selects an element with the attribute ''foo'' whose value is ''bar''
* '''foo bar''' selects an element matching ''bar'' that is a descendant of an element matching ''foo''
* '''foo > bar''' selects an element matching ''car'' that is a child of an element matching ''foo''
* '''foo + bar''' selects an element matching ''bar'' that is the next sibling of an element matching ''foo''

==== Pseudo-Classes ====

Elements can also be selected based on their current state using ''pseudo-classes''. Pseudo-classes always begin with a colon. For example, this is how you could change the color of all visited links on your web page:

<source lang="css">
a:visited {
color: #99FF99; /* this example uses a hexadecimal RGB definition for the color, which allows you to
mix over 16 million colors */
}
</source>

Some common pseudo-classes are listed below. For a complete list, see [https://www.w3.org/TR/CSS/#selectors section 4.2 of the CSS Specification].

* '''foo:first-child''' selects an element matching ''foo'' that is the first child of its parent (i.e., any ''foo'' that doesn't have older siblings)
* '''foo:last-child''' selects an element matching ''foo'' that is the last child of its parent
* '''foo:empty''' selects any element matching ''foo'' that has no children
* Pseudo-Classes for Form Elements:
** '''input:checked''' selects all checkboxes and radio buttons that are checked
* Pseudo-Classes for Hyperlink Elements:
** '''a:link''' selects all hyperlinks that have not been visited yet
** '''a:visited''' selects all hyperlinks that have been visited
* Pseudo-Classes for Interactive Elements (like buttons, hyperlinks, etc):
** '''foo:active''' selects elements matching ''foo'' that are being pressed on
** '''foo:hover''' selects elements matching ''foo'' that are being hovered over
** '''foo:focus''' selects elements matching ''foo'' that are selected (no pun intended)

=== Properties ===

Selectors are used to select elements that need to have styles applied to them, and properties are used to actually define those styles. Throughout this section, you've seen properties including '''color''', '''font-family''', '''font-style''', and '''text-decoration'''. There are dozens more properties you can define, some of which are listed below.

For a complete list of CSS properties, see [https://www.w3.org/TR/CSS/#properties section 4.1 of the CSS Specification].

* '''background-image''' sets the background image of an element. Usage: <code>background-image: url(path/to/image.png);</code>
* '''background-color''' sets the background color of an element (note: the image takes precedence over the color; the color will be displayed if the image can't be found)
* For Tables:
** '''border-collapse''' removes the spacing between table cells. Usage: <code>border-collapse: collapse;</code>
** Note: You can select td elements that are grandchildren of a table element, and then specify border properties on the td's, like this: <code>#myTable > tr > td { border: 1px solid black; }</code>
* '''border''' defines the border around an element. Usage: <code>border: 1px dashed red;</code>
* '''border-radius''' lets you have rounded corners on your elements
* '''float''' makes the text of a document flow around the element. Usage: <code>float: left;</code> (or <code>right</code>)
** '''clear''' makes an element stop wrapping around floats and instead move down to where the floats end. Usage: <code>clear: both;</code>
** Note: If you specify multiple object in a for as floating, they will appear next to each other horizontally.
* '''cursor''' lets you change the type of cursor that appears then your element is in focus. Example usage: <code>cursor: pointer;</code>
* '''font''' lets you specify the font-size, line-height, and font-family in one fell swoop. Usage: <code>font: 14px/18px Verdana, Arial, sans-serif;</code>
* '''list-style-type''' specifies the type of number or bullet used on <code><nowiki><ol></nowiki></code>'s and <code><nowiki><ul></nowiki></code>'s
* '''text-align''' lets you justify text on the left, center, or right
* '''display''' lets us specify an element as inline or block (see below), and it also enables us to "turn off" an element from being displayed on a page. Example usage: <code>display: none;</code>

==== Units ====

Some CSS properties require that you specify numerical values for things (font-size, line-height, border-width, and so on). CSS supports a number of units, the most common of which include:

* '''px''', which defines a distance in pixels (relative to screen resolution).
* '''em''', which defines a distance relative to font size. (Note: an ''em'' is the width of the capital letter M. No, I'm not kidding.)
* '''rem''', which defines a distance relative to the font size of the root element. Compare to '''em''', which defines a distance relative to the font size of the current element (which might have a different font size than the document root).
* '''pt''', which is a traditional print unit equal to 1/72 of an inch (should appear the same size on all screens regardless of resolution).
* '''%''', which defines a distance relative to that property of its parent element. For the most part, you'll only use % when working with the Box Model (see below).

=== The Box Model ===

For the most part, all elements fall into one of two categories:

* Inline (Examples: a, span, strong)
* Block (Examples: p, div, section)

The properties we've seen thus far apply mostly to inline elements. As it turns out, CSS has an additional set of properties that describe block elements. These properties compose the ''box model''.

While inline elements flow with the text, block elements stand on their own. You can define the space between the block element and the next closest element on the page using ''margin'', the block element's border using ''border'', and the space between the block element's border and its content using ''padding'':

[[File:Box.png]]

For example, you could make a padded box that is centered on the page using this CSS:

<source lang="css">
div.box {
width: 500px;
margin: 0 auto; /* when two "arguments" are passed to margin or padding, the first defines top
and bottom margin/padding and the second defines left and right margin/padding */
padding: 25px; /* when only one "argument" is passed to margin or padding, it defines the
margin/padding on all four sides */
border: 1px solid black; /* the 1px defines the border-width; the solid defines the border-style;
the black defines the border-color */
}
</source>

Here is what the resulting box looks like:

[[File:CSSBoxExample.png]]

Note that the ''width'' defines only the ''content width''. To compute the actual "width" of the element, you need to add the left and right padding border, and margin.

This is only scraping the surface of the CSS box model. All advanced web sites utilize the box model to its full extent to produce the complex layouts you have come to expect on the internet. For more information on the CSS box model, read [https://www.w3.org/TR/css3-box/ the W3C specification].

=== Typography ===

Traditionally, Web developers were limited to a small subset of fonts that were commonly installed on all computers:

* Andale Mono
* Arial
* Arial Black
* Comic Sans MS
* Courier New
* Georgia
* Impact
* Times New Roman
* Trebuchet MS
* Verdana
* Webdings

That was it. Incredibly limiting.

However, fonts other than these are used widely on the Web. How? CSS has evolved to the point where it now allows you to upload a font to your server, and use that font on your site, or even include CSS from another server that adds the fonts you want. One excellent resource for a wide variety of fast, appropriately-licensed fonts is [https://fonts.google.com/ Google Fonts].

M1 Introduction to HTML

2023-08-05T01:20:08Z

Zach: /* HTML */

== HTML ==

=== HTML? ===

HTML, which stands for HyperText Markup Language, is the standard language used to create webpages. It describes the structure and content of a webpage. HTML uses a system of tags and attributes to define elements on the page. These elements can be things like headings, paragraphs, links, images, lists, tables, forms, and many more. Each of these elements has a specific tag that describes what it is and helps the browser know how to display it.

Here's a very basic example:

<source lang="html">
<!DOCTYPE html>
<html>
<head>
<title>My First Web Page</title>
</head>
<body>
<h1>Welcome to My Website!</h1>
<p>This is a paragraph of text.</p>
</body>
</html>
</source>

'''Image will go here'''

In this example:

* <source lang="html" enclose="none"><!DOCTYPE html></source> The page begins with a DOCTYPE declaration. This is a mandatory part of a web page, and it tells the browser what version of HTML to use when rendering your page. In this case, the browser should use HTML 5.
* The <source lang="html" enclose="none"><html></source> tag is the root of an HTML document.
* The <source lang="html" enclose="none"><head></source> tag contains meta-information about the document. This can include the title, character set declarations, CSS style definitions, and more.
* The <source lang="html" enclose="none"><title></source> tag is the title of the document and is displayed in the browser's title bar or tab.
* The <source lang="html" enclose="none"><body></source> tag contains the content of the document, such as text, hyperlinks, images, tables, lists, etc. When you visit a website, the content you see is rendered in the body.
* The <source lang="html" enclose="none"><h1></source> tag defines a top-level heading, and the <source lang="html" enclose="none"><p></source> tag defines a paragraph.

In essence, HTML gives the webpage its basic structure and content; without it, a webpage can't exist.

''You can view a web page's source code in your browser; in Firefox, the keyboard shortcut is Ctrl-U (or Command-U), and in Chrome, it is Alt-Ctrl-U (or Option-Command-U).''

=== Tags ===

Tags in HTML are special words or symbols enclosed in angle brackets (< >) that define the structure and formatting of elements within a web page. They act as instructions to the web browser on how to display and interpret the content. Opening tags, such as <tagname>, mark the beginning of an element, while closing tags, like </tagname>, denote the end. Tags are like containers that hold different types of content, such as text, images, links, or other elements, and they play a crucial role in creating the structure and appearance of a webpage.

Here are some of the most common HTML tags you will encounter. Get to know these well. For more information on how to use a given tag, [https://developer.mozilla.org/en-US/docs/Web/HTML/Element|the MDN docs] are an incredible resource.

{| class="wikitable"
|-
! style="width:15%" | Tag
! style="width:85%" | Description
|-
| <source lang="html" enclose="none"><html></source>
| Defines the root element of an HTML document.
|-
| <source lang="html" enclose="none"><head></source>
| Contains metadata and information about the HTML document. (docs)
|-
| <source lang="html" enclose="none"><title></source>
| Sets the title of the document, displayed in the browser's title bar or tab. (docs)
|-
| <source lang="html" enclose="none"><body></source>
| Represents the main content of the HTML document.
|-
| <source lang="html" enclose="none"><h1></source> to <source lang="html"enclose="none"><h6></source>
| Defines headings of different levels, from the most important (<source lang="html" enclose="none"><h1></source>) to the least (<source lang="html" enclose="none"><h6></source>).
|-
| <source lang="html" enclose="none"><p></source>
| Represents a paragraph of text.
|-
| <source lang="html" enclose="none"><a></source>
| Creates a hyperlink to another web page, file, or a specific location within the same page.
|-
| <source lang="html" enclose="none"><img></source>
| Inserts an image into the HTML document.
|-
| <source lang="html" enclose="none"><ul></source>
| Represents an unordered list (a bulleted list).
|-
| <source lang="html" enclose="none"><ol></source>
| Represents an ordered list (a numbered list).
|-
| <source lang="html" enclose="none"><li></source>
| Defines an item in a list (<source lang="html" enclose="none"><ul></source> or <source lang="html" enclose="none"><ol></source>).
|-
| <source lang="html" enclose="none"><table></source>
| Represents tabular data organized into rows and columns.
|-
| <source lang="html" enclose="none"><tr></source>
| Defines a row within a table.
|-
| <source lang="html" enclose="none"><td></source>
| Defines a cell within a table row (<source lang="html" enclose="none"><tr></source>).
|-
| <source lang="html" enclose="none"><th></source>
| Defines a header cell within a table row (<source lang="html" enclose="none"><tr></source>).
|-
| <source lang="html" enclose="none"><div></source>
| Defines a division or a container for other elements.
|-
| <source lang="html" enclose="none"><span></source>
| Defines an inline container for a small section of HTML.
|-
| <source lang="html" enclose="none"><form></source>
| Creates an interactive form for user input.
|-
| <source lang="html" enclose="none"><input></source>
| Creates an input field for user data entry.
|-
| <source lang="html" enclose="none"><button></source>
| Creates a clickable button.
|-
| <source lang="html" enclose="none"><textarea></source>
| Creates a multiline text input field.
|}

=== Built-in Functionality ===

While HTML primarily focuses on structure and content, some HTML tags come with built-in functionality that extends beyond that. These tags incorporate specific behaviors or interactivity into the elements they define. For example, the <source lang="html" enclose="none"><a></source> tag creates hyperlinks, allowing users to navigate to different web pages or sections within the same page. The <source lang="html" enclose="none"><form></source> tag enables the creation of interactive forms to gather user input, and the <source lang="html" enclose="none"><input></source> tag provides various types of input fields like text boxes or checkboxes.

To learn about less common tags with fun built-in functionality, see the optional reading, [[M1 HTML Extended|HTML Extended]]. I also encourage you to spend some time exploring [https://developer.mozilla.org/en-US/docs/Web/HTML/Element|the MDN docs].

=== Default Styling ===

When rendering HTML documents, web browsers apply default styles to each HTML element. These default styles determine how the elements are displayed visually, such as their font, color, size, spacing, and positioning. However, it's important to note that default styles may vary slightly between different browsers.

Browsers have their own predefined stylesheets, often referred to as user agent stylesheets, which dictate the initial appearance of HTML elements. This means that even without explicitly applying any styles, elements will still have some visual presentation based on these default styles.

See [M1 Introduction to CSS|Introduction to CSS] for a guide to styling elements.

=== Nodes ===

Where a tag is a syntax used in HTML to define elements, a node is a conceptual representation of an element within the Document Object Model (DOM) hierarchy. In simpler terms, a tag is the markup language used to define an element, whereas a node is an object that represents that element within the structured model of the HTML document.

Nodes form a hierarchical structure known as the Document Object Model (DOM). The DOM represents the HTML document as a tree-like structure, where each node has a relationship with its parent, children, and siblings. The topmost node in the DOM is the document node, representing the entire HTML document.

Here are the main types of nodes commonly encountered in HTML:

==== Element Nodes ====

These nodes represent HTML elements and are the most common type. They correspond to the opening and closing tags of an HTML element. For example, <source lang="html" enclose="none"><div>...</div></source>, <source lang="html" enclose="none"><p>...</p></source>, <source lang="html" enclose="none"><h1>...</h1></source> are all element nodes. Element nodes can contain child nodes, such as other elements, text nodes, or comment nodes.

==== Text Nodes ====

Text nodes represent the textual content within an HTML element. They can contain plain text, including spaces, line breaks, or any other characters. For instance, in the element

<source lang="html"><p>Hello, <strong>world!</strong></p></source>

the text node would contain "Hello," and "world!".

==== Attribute Nodes ====

These nodes represent the attributes of an HTML element. An element can have multiple attributes like id, class, src, etc. Attribute nodes contain the name and value of the attribute. For example, in the element

<source lang="html"><img src="image.jpg" alt="An image" /></source>

the `src` and `alt` attributes would be represented as attribute nodes.

==== Comment Nodes ====

Comment nodes represent HTML comments within the document. They are used to add explanatory or descriptive notes that are not displayed in the rendered output. Comment nodes start with <source lang="html" enclose="none"></source>. For example:

<source lang="html"></source>

=== The High School Analogy ===
In more ways than not, HTML can be thought of as a markup language for five-paragraph essays that you wrote in history class in high school. <source lang="html" enclose="none"><blockquote></source> is used to denote long quotations; <source lang="html" enclose="none"><h1></source> through <source lang="html" enclose="none"><h6></source> denote the outline; <source lang="html" enclose="none"><p></source> denotes new paragraphs; <source lang="html" enclose="none"><figure></source> and <source lang="html" enclose="none"><figcaption></source> represent exhibits; <source lang="html" enclose="none"><section></source> could represent chapters if you were writing a longer report; and so on.
Take home message: HTML defines the content of your page. Use it to its full ability.

M1 Introduction to HTML

2023-08-05T01:14:08Z

Zach: /* HTML? */

M1 Introduction to HTML

2023-06-27T04:44:33Z

Zach: /* Built-in Functionality */

== HTML ==

=== HTML? ===

HTML, which stands for HyperText Markup Language, is the standard language used to create webpages. It describes the structure and content of a webpage. HTML uses a system of tags and attributes to define elements on the page. These elements can be things like headings, paragraphs, links, images, lists, tables, forms, and many more. Each of these elements has a specific tag that describes what it is and helps the browser know how to display it.

Here's a very basic example:

<source lang="html">
<!DOCTYPE html>
<html>
<head>
<title>My First Web Page</title>
</head>
<body>
<h1>Welcome to My Website!</h1>
<p>This is a paragraph of text.</p>
</body>
</html>
</source>

'''Image will go here'''

In this example:

* <source lang="html" enclose="none"><!DOCTYPE html></source> is the document type declaration. This tells the browser that this document is an HTML document.
* The <source lang="html" enclose="none"><html></source> tag is the root of an HTML document.
* The <source lang="html" enclose="none"><head></source> tag contains meta-information about the document. This can include the title, character set declarations, CSS style definitions, and more.
* The <source lang="html" enclose="none"><title></source> tag is the title of the document and is displayed in the browser's title bar or tab.
* The <source lang="html" enclose="none"><body></source> tag contains the content of the document, such as text, hyperlinks, images, tables, lists, etc. When you visit a website, the content you see is rendered in the body.
* The <source lang="html" enclose="none"><h1></source> tag defines a top-level heading, and the <source lang="html" enclose="none"><p></source> tag defines a paragraph.

In essence, HTML gives the webpage its basic structure and content; without it, a webpage can't exist.

=== Tags ===

Tags in HTML are special words or symbols enclosed in angle brackets (< >) that define the structure and formatting of elements within a web page. They act as instructions to the web browser on how to display and interpret the content. Opening tags, such as <tagname>, mark the beginning of an element, while closing tags, like </tagname>, denote the end. Tags are like containers that hold different types of content, such as text, images, links, or other elements, and they play a crucial role in creating the structure and appearance of a webpage.

Here are some of the most common HTML tags you will encounter. Get to know these well. For more information on how to use a given tag, [https://developer.mozilla.org/en-US/docs/Web/HTML/Element|the MDN docs] are an incredible resource.

{| class="wikitable"
|-
! style="width:15%" | Tag
! style="width:85%" | Description
|-
| <source lang="html" enclose="none"><html></source>
| Defines the root element of an HTML document.
|-
| <source lang="html" enclose="none"><head></source>
| Contains metadata and information about the HTML document. (docs)
|-
| <source lang="html" enclose="none"><title></source>
| Sets the title of the document, displayed in the browser's title bar or tab. (docs)
|-
| <source lang="html" enclose="none"><body></source>
| Represents the main content of the HTML document.
|-
| <source lang="html" enclose="none"><h1></source> to <source lang="html"enclose="none"><h6></source>
| Defines headings of different levels, from the most important (<source lang="html" enclose="none"><h1></source>) to the least (<source lang="html" enclose="none"><h6></source>).
|-
| <source lang="html" enclose="none"><p></source>
| Represents a paragraph of text.
|-
| <source lang="html" enclose="none"><a></source>
| Creates a hyperlink to another web page, file, or a specific location within the same page.
|-
| <source lang="html" enclose="none"><img></source>
| Inserts an image into the HTML document.
|-
| <source lang="html" enclose="none"><ul></source>
| Represents an unordered list (a bulleted list).
|-
| <source lang="html" enclose="none"><ol></source>
| Represents an ordered list (a numbered list).
|-
| <source lang="html" enclose="none"><li></source>
| Defines an item in a list (<source lang="html" enclose="none"><ul></source> or <source lang="html" enclose="none"><ol></source>).
|-
| <source lang="html" enclose="none"><table></source>
| Represents tabular data organized into rows and columns.
|-
| <source lang="html" enclose="none"><tr></source>
| Defines a row within a table.
|-
| <source lang="html" enclose="none"><td></source>
| Defines a cell within a table row (<source lang="html" enclose="none"><tr></source>).
|-
| <source lang="html" enclose="none"><th></source>
| Defines a header cell within a table row (<source lang="html" enclose="none"><tr></source>).
|-
| <source lang="html" enclose="none"><div></source>
| Defines a division or a container for other elements.
|-
| <source lang="html" enclose="none"><span></source>
| Defines an inline container for a small section of HTML.
|-
| <source lang="html" enclose="none"><form></source>
| Creates an interactive form for user input.
|-
| <source lang="html" enclose="none"><input></source>
| Creates an input field for user data entry.
|-
| <source lang="html" enclose="none"><button></source>
| Creates a clickable button.
|-
| <source lang="html" enclose="none"><textarea></source>
| Creates a multiline text input field.
|}

=== Built-in Functionality ===

While HTML primarily focuses on structure and content, some HTML tags come with built-in functionality that extends beyond that. These tags incorporate specific behaviors or interactivity into the elements they define. For example, the <source lang="html" enclose="none"><a></source> tag creates hyperlinks, allowing users to navigate to different web pages or sections within the same page. The <source lang="html" enclose="none"><form></source> tag enables the creation of interactive forms to gather user input, and the <source lang="html" enclose="none"><input></source> tag provides various types of input fields like text boxes or checkboxes.

To learn about less common tags with fun built-in functionality, see the optional reading, [[M1 HTML Extended|HTML Extended]]. I also encourage you to spend some time exploring [https://developer.mozilla.org/en-US/docs/Web/HTML/Element|the MDN docs].

=== Default Styling ===

When rendering HTML documents, web browsers apply default styles to each HTML element. These default styles determine how the elements are displayed visually, such as their font, color, size, spacing, and positioning. However, it's important to note that default styles may vary slightly between different browsers.

Browsers have their own predefined stylesheets, often referred to as user agent stylesheets, which dictate the initial appearance of HTML elements. This means that even without explicitly applying any styles, elements will still have some visual presentation based on these default styles.

See [M1 Introduction to CSS|Introduction to CSS] for a guide to styling elements.

=== Nodes ===

Where a tag is a syntax used in HTML to define elements, a node is a conceptual representation of an element within the Document Object Model (DOM) hierarchy. In simpler terms, a tag is the markup language used to define an element, whereas a node is an object that represents that element within the structured model of the HTML document.

Nodes form a hierarchical structure known as the Document Object Model (DOM). The DOM represents the HTML document as a tree-like structure, where each node has a relationship with its parent, children, and siblings. The topmost node in the DOM is the document node, representing the entire HTML document.

Here are the main types of nodes commonly encountered in HTML:

==== Element Nodes ====

These nodes represent HTML elements and are the most common type. They correspond to the opening and closing tags of an HTML element. For example, <source lang="html" enclose="none"><div>...</div></source>, <source lang="html" enclose="none"><p>...</p></source>, <source lang="html" enclose="none"><h1>...</h1></source> are all element nodes. Element nodes can contain child nodes, such as other elements, text nodes, or comment nodes.

==== Text Nodes ====

Text nodes represent the textual content within an HTML element. They can contain plain text, including spaces, line breaks, or any other characters. For instance, in the element

<source lang="html"><p>Hello, <strong>world!</strong></p></source>

the text node would contain "Hello," and "world!".

==== Attribute Nodes ====

These nodes represent the attributes of an HTML element. An element can have multiple attributes like id, class, src, etc. Attribute nodes contain the name and value of the attribute. For example, in the element

<source lang="html"><img src="image.jpg" alt="An image" /></source>

the `src` and `alt` attributes would be represented as attribute nodes.

==== Comment Nodes ====

Comment nodes represent HTML comments within the document. They are used to add explanatory or descriptive notes that are not displayed in the rendered output. Comment nodes start with <source lang="html" enclose="none"></source>. For example:

<source lang="html"></source>

M1 Introduction to HTML

2023-06-27T04:41:27Z

Zach: /* HTML */

== HTML ==

=== HTML? ===

HTML, which stands for HyperText Markup Language, is the standard language used to create webpages. It describes the structure and content of a webpage. HTML uses a system of tags and attributes to define elements on the page. These elements can be things like headings, paragraphs, links, images, lists, tables, forms, and many more. Each of these elements has a specific tag that describes what it is and helps the browser know how to display it.

Here's a very basic example:

<source lang="html">
<!DOCTYPE html>
<html>
<head>
<title>My First Web Page</title>
</head>
<body>
<h1>Welcome to My Website!</h1>
<p>This is a paragraph of text.</p>
</body>
</html>
</source>

'''Image will go here'''

In this example:

* <source lang="html" enclose="none"><!DOCTYPE html></source> is the document type declaration. This tells the browser that this document is an HTML document.
* The <source lang="html" enclose="none"><html></source> tag is the root of an HTML document.
* The <source lang="html" enclose="none"><head></source> tag contains meta-information about the document. This can include the title, character set declarations, CSS style definitions, and more.
* The <source lang="html" enclose="none"><title></source> tag is the title of the document and is displayed in the browser's title bar or tab.
* The <source lang="html" enclose="none"><body></source> tag contains the content of the document, such as text, hyperlinks, images, tables, lists, etc. When you visit a website, the content you see is rendered in the body.
* The <source lang="html" enclose="none"><h1></source> tag defines a top-level heading, and the <source lang="html" enclose="none"><p></source> tag defines a paragraph.

In essence, HTML gives the webpage its basic structure and content; without it, a webpage can't exist.

=== Tags ===

Tags in HTML are special words or symbols enclosed in angle brackets (< >) that define the structure and formatting of elements within a web page. They act as instructions to the web browser on how to display and interpret the content. Opening tags, such as <tagname>, mark the beginning of an element, while closing tags, like </tagname>, denote the end. Tags are like containers that hold different types of content, such as text, images, links, or other elements, and they play a crucial role in creating the structure and appearance of a webpage.

Here are some of the most common HTML tags you will encounter. Get to know these well. For more information on how to use a given tag, [https://developer.mozilla.org/en-US/docs/Web/HTML/Element|the MDN docs] are an incredible resource.

{| class="wikitable"
|-
! style="width:15%" | Tag
! style="width:85%" | Description
|-
| <source lang="html" enclose="none"><html></source>
| Defines the root element of an HTML document.
|-
| <source lang="html" enclose="none"><head></source>
| Contains metadata and information about the HTML document. (docs)
|-
| <source lang="html" enclose="none"><title></source>
| Sets the title of the document, displayed in the browser's title bar or tab. (docs)
|-
| <source lang="html" enclose="none"><body></source>
| Represents the main content of the HTML document.
|-
| <source lang="html" enclose="none"><h1></source> to <source lang="html"enclose="none"><h6></source>
| Defines headings of different levels, from the most important (<source lang="html" enclose="none"><h1></source>) to the least (<source lang="html" enclose="none"><h6></source>).
|-
| <source lang="html" enclose="none"><p></source>
| Represents a paragraph of text.
|-
| <source lang="html" enclose="none"><a></source>
| Creates a hyperlink to another web page, file, or a specific location within the same page.
|-
| <source lang="html" enclose="none"><img></source>
| Inserts an image into the HTML document.
|-
| <source lang="html" enclose="none"><ul></source>
| Represents an unordered list (a bulleted list).
|-
| <source lang="html" enclose="none"><ol></source>
| Represents an ordered list (a numbered list).
|-
| <source lang="html" enclose="none"><li></source>
| Defines an item in a list (<source lang="html" enclose="none"><ul></source> or <source lang="html" enclose="none"><ol></source>).
|-
| <source lang="html" enclose="none"><table></source>
| Represents tabular data organized into rows and columns.
|-
| <source lang="html" enclose="none"><tr></source>
| Defines a row within a table.
|-
| <source lang="html" enclose="none"><td></source>
| Defines a cell within a table row (<source lang="html" enclose="none"><tr></source>).
|-
| <source lang="html" enclose="none"><th></source>
| Defines a header cell within a table row (<source lang="html" enclose="none"><tr></source>).
|-
| <source lang="html" enclose="none"><div></source>
| Defines a division or a container for other elements.
|-
| <source lang="html" enclose="none"><span></source>
| Defines an inline container for a small section of HTML.
|-
| <source lang="html" enclose="none"><form></source>
| Creates an interactive form for user input.
|-
| <source lang="html" enclose="none"><input></source>
| Creates an input field for user data entry.
|-
| <source lang="html" enclose="none"><button></source>
| Creates a clickable button.
|-
| <source lang="html" enclose="none"><textarea></source>
| Creates a multiline text input field.
|}

=== Built-in Functionality ===

While HTML primarily focuses on structure and content, some HTML tags come with built-in functionality that extends beyond that. These tags incorporate specific behaviors or interactivity into the elements they define. For example, the <nowiki><a></nowiki> tag creates hyperlinks, allowing users to navigate to different web pages or sections within the same page. The <nowiki><form></nowiki> tag enables the creation of interactive forms to gather user input, and the <nowiki><input></nowiki> tag provides various types of input fields like text boxes or checkboxes.

To learn about less common tags with fun built-in functionality, see the optional reading, [[HTML Extended|../optional/HTML_extended.md]]. I also encourage you to spend some time exploring [the MDN docs](https://developer.mozilla.org/en-US/docs/Web/HTML/Element).

=== Default Styling ===

When rendering HTML documents, web browsers apply default styles to each HTML element. These default styles determine how the elements are displayed visually, such as their font, color, size, spacing, and positioning. However, it's important to note that default styles may vary slightly between different browsers.

Browsers have their own predefined stylesheets, often referred to as user agent stylesheets, which dictate the initial appearance of HTML elements. This means that even without explicitly applying any styles, elements will still have some visual presentation based on these default styles.

See [M1 Introduction to CSS|Introduction to CSS] for a guide to styling elements.

=== Nodes ===

Where a tag is a syntax used in HTML to define elements, a node is a conceptual representation of an element within the Document Object Model (DOM) hierarchy. In simpler terms, a tag is the markup language used to define an element, whereas a node is an object that represents that element within the structured model of the HTML document.

Nodes form a hierarchical structure known as the Document Object Model (DOM). The DOM represents the HTML document as a tree-like structure, where each node has a relationship with its parent, children, and siblings. The topmost node in the DOM is the document node, representing the entire HTML document.

Here are the main types of nodes commonly encountered in HTML:

==== Element Nodes ====

These nodes represent HTML elements and are the most common type. They correspond to the opening and closing tags of an HTML element. For example, <source lang="html" enclose="none"><div>...</div></source>, <source lang="html" enclose="none"><p>...</p></source>, <source lang="html" enclose="none"><h1>...</h1></source> are all element nodes. Element nodes can contain child nodes, such as other elements, text nodes, or comment nodes.

==== Text Nodes ====

Text nodes represent the textual content within an HTML element. They can contain plain text, including spaces, line breaks, or any other characters. For instance, in the element

<source lang="html"><p>Hello, <strong>world!</strong></p></source>

the text node would contain "Hello," and "world!".

==== Attribute Nodes ====

These nodes represent the attributes of an HTML element. An element can have multiple attributes like id, class, src, etc. Attribute nodes contain the name and value of the attribute. For example, in the element

<source lang="html"><img src="image.jpg" alt="An image" /></source>

the `src` and `alt` attributes would be represented as attribute nodes.

==== Comment Nodes ====

Comment nodes represent HTML comments within the document. They are used to add explanatory or descriptive notes that are not displayed in the rendered output. Comment nodes start with <source lang="html" enclose="none"></source>. For example:

<source lang="html"></source>

M1 Introduction to HTML

2023-06-27T04:31:49Z

Zach: /* Tags */

M1 Introduction to HTML

2023-06-27T04:26:32Z

Zach: /* Tags */

M1 Introduction to HTML

2023-06-27T04:21:15Z

Zach: /* Tags */

M1 Introduction to HTML

2023-06-27T04:05:12Z

Zach: /* HTML? */

M1 Introduction to HTML

2023-06-27T03:46:45Z

Zach: Created page with "== HTML == === HTML? === HTML, which stands for HyperText Markup Language, is the standard language used to create webpages. It describes the structure and content of a webp..."

M1 Reading

2023-06-27T03:27:00Z

Zach: Created page with "== Module 1 | Reading == '''Students should read each page listed under the "Required" heading. These texts will be critical for completing this module's assignment(s).'''..."

== Module 1 | Reading ==

'''Students should read each page listed under the "Required" heading. These texts will be critical for completing this module's assignment(s).'''

Texts listed as "Supplemental" are highly recommended.

* Required
** [[M1 Introduction to HTML|Introduction to HTML]]
** [[M1 Introduction to CSS|Introduction to CSS]]
** [[M1 Introduction to Git|Introduction to Git]]
* Supplemental
** [[M1 Markdown|Markdown]]
** [[M1 Accessibility|Accessibility]]
** [[M1 HTML Extended|HTML Extended]]
** [[M1 CSS Extended|CSS Extended]]
** [[M1 Git Extended|Git Extended]]

Rewrite 2023

2023-06-27T03:23:10Z

Zach: Created page with "== 2023 Wiki reorg == I am working on a rewrite/reorg of the current wiki. In its current state, the wiki has an incredible amount of information, but I've heard from many stu..."

== 2023 Wiki reorg ==
I am working on a rewrite/reorg of the current wiki. In its current state, the wiki has an incredible amount of information, but I've heard from many students that it is difficult to parse and sometimes confusing as to which data is relevant. This effort serves to address some of these issues.

In its current state, students often find it difficult to separate the assignment-critical info from supplementary content. To address this, the rewrite seperates structures each assignment like this:

* Purpose (of module)
* Assignemnt
** Individual
** Group

* Reading
** Required
** Optional

=== Directory ===
* [[M1 Index|M1]]
** [[M1 Purpose|Purpose]]
** [[M1 Individual|Individual Assignment]]
** [[M1 Reading|Reading]]
*** Required
**** [[M1 Introduction to HTML|Introduction to HTML]]
**** [[M1 Introduction to CSS|Introduction to CSS]]
**** [[M1 Introduction to Git|Introduction to Git]]
*** Supplemental
**** [[M1 Markdown|Markdown]]
**** [[M1 Accessibility|Accessibility]]
**** [[M1 HTML Extended|HTML Extended]]
**** [[M1 CSS Extended|CSS Extended]]
**** [[M1 Git Extended|Git Extended]]

<blockquote> ''More to be added soon. The other modules will follow the same basic structure.''</blockquote>

CSE 330 Online Textbook - Table of Contents

2023-02-07T19:20:31Z

Zach:

<div style="text-align:center;">[[File:CSE330Banner.png]]</div>

__NOTOC__
<div style="column-width: 300px; -moz-column-width: 300px; -webkit-column-width: 300px;">
{{FrontPageModule
|num=1
|subject=HTML and CSS Fundamentals
|links=
* [[HTML and CSS]]
* [[HTML Forms]]
* [[Git]]
* [[FAQ - Mod 1]]
}}

{{FrontPageModule
|num=2
|subject=Linux, Apache, and PHP
|links=
* [[Linux]]
* [[Bash]]
* [[Amazon Web Services]]
* [[SSH]]
* [[Apache]]
* [[PHP]]
* [[Workflow]]
* [[VS Code]]
* [[Web Application Security, Part 1]]
* [[FAQ - Mod 2]]
}}

{{FrontPageModule
|num=3
|subject=SQL Databases
|links=
* [[Introduction to MySQL]] (installation)
* [[MySQL Schema and State]] (queries)
* [[PHP and MySQL]] (using MySQL from PHP)
* [[Web Application Security, Part 2]]
* [[FAQ - Mod 3]]
}}

{{FrontPageModule
|num=4
|subject=Scripting for Rapid Prototyping
|links=
* [[Python]]
* [[Regular Expressions]]
* [[FAQ - Mod 4]]
}}

{{FrontPageModule
|num=5
|subject=JavaScript and APIs
|links=
* [[JavaScript]]
* [[AJAX and JSON]]
* [[Web Application Security, Part 3]]
* [[FAQ - Mod 5]]

}}

{{FrontPageModule
|num=6
|subject=Real-Time Web Applications
|links=
* [[Node.JS]]
* [[Socket.IO]]
* [[FAQ - Mod 6]]
}}

{{FrontPageModule
|num=7
|subject=Creative Project
|links=
* [[Choosing a Platform]]
* [[Web Frameworks]]
}}

</div>

== Reference ==

Use this wiki as a starting place for your projects in CSE 330. However, web development is a large field, so it is impossible for us to document everything you might need to know. Here are some other places you should look when you need to find something:

* [https://developer.mozilla.org/en-US/ The Mozilla Development Network (MDN)], a community-driven source containing documentation and tutorials for Web technologies
* [http://reference.sitepoint.com/html SitePoint], which has a reference page as well as quality articles and tutorials

If you can't find what you need in any of these places, try using your favorite search engine. '''Be wary of sites that come up on search engines.''' Many HTML reference sites document bad practices and dated techniques. If you're unsure whether a reference site you find is legitimate, ask a TA.

== Want to Learn More? ==

The limited amount of time in CSE 330 means that we can cover only a subset of the wide range of possible topics in the field of rapid prototyping and web development. Indeed, the exact topics covered in CSE 330 change from year to year.

Feel free to browse some of the topics that have previously been covered in CSE 330: [[:Category:Former Content|CSE 330 Former Content]]

You should also consider following blogs like [http://www.sitepoint.com/ SitePoint] to keep up-to-date on the latest developments in the field of web development.

== Content License ==

The content of this wiki is copyright (c) 2013, Washington University in St. Louis, Department of Computer Science. All rights reserved.

=== Modified BSD License ===

Redistribution and use in source and binary forms, with or without modification, by a Third Party who has not completed and is not in the process of completing CSE 330, of the code examples or concepts on this wiki, are permitted, provided that the following conditions are met:

* Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
* Neither the name of the Washington University School of Engineering and Applied Science nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS TO ALL PARTIES "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

=== What it Means ===

If you are a student in CSE 330, you may use the code from this wiki freely in your projects without the need for attribution. If you are not a student in CSE 330, you may use the code from this wiki freely in your projects with attribution. Either way, the code examples are provided "as is", without any warranty, and Washington University holds no liability for problems arising from the use of the content in this wiki.

PHP

2023-02-07T19:18:04Z

Zach: /* Sessions */

PHP is an open-source language used widely for writing server-side code for web servers. PHP is a good starting point for web developers, as it is straightforward to learn and yet powerful at the same time. Indeed, some of the internet's most popular web sites (including Facebook and Wikipedia) are written using PHP as their language of choice! This article introduces the syntax of PHP and how to make a basic PHP web application.

{{XKCD
|name=mac_pc
|id=934
}}

== What is PHP? ==

PHP is a server-side scripting language widely used in web development. PHP is what turns your web site from a static fileserver into a dynamic web application.

By default, Apache only knows how to serve static HTML pages to the client. You can write as many HTML pages as you want, which is fine for basic web sites. But if you want to have users post to a forum, or process credit card transactions, or make your own online calendar, a server-side language like PHP is what you need.

=== Document Generation Language ===

If PHP's original core purpose had to be described in three words, it would be this: '''document generation language'''. Unlike languages including Java, Ruby, Python, and Perl, which consist of classes in a strict program layout, PHP code is most often embedded inside a template document. When the PHP interpreter runs, it "fills in the gaps" and generates a finished product document.

This isn't to say that PHP can ''only'' be a document generating language, because this is certainly not true. However, for the purposes of Module 2, it is safe to think of it this way when you are learning the concepts behind a web server.

==== MIME Types ====

By default, PHP generates HTML documents. (Recall that an HTML document is simply a block of text consisting of markup conforming to the HTML standard.)

However, PHP does not need to generate HTML documents. It can also generate images, PDFs, or any other type of document. Each of these document types has its own unique markup language. The markup language used by a certain document is defined by its ''MIME Type''.

The HTML MIME Type is '''text/html'''. Other MIME Types include '''image/jpeg''', '''image/png''', and '''application/xml'''.

To change the MIME Type of the document generated by PHP, use the statement:

<source lang="php">
header("Content-Type: image/jpeg");
</source>

Since PHP generates HTML documents by default, you can think of there being an implicit <code>header("Content-Type: text/html");</code> at the top of every PHP script.

=== PHP and Apache ===

When you request a PHP file over Apache, the file is ''not'' sent verbatim. Instead, it is first sent through the PHP interpreter. Here is an oversimplified schematic of the process:

[[File:PHP-Apache_Flowchart.png]]

In summary, here are some points to keep in mind.

# A PHP file on your server is not an HTML document. It is a PHP script.
# PHP generates documents. By default, it generates HTML documents.
# PHP can generate other document types by changing the MIME Type in the Content-Type HTTP Header.
# Apache runs the PHP interpreter on your PHP script before transmitting it to the browser. The content that is generated after the PHP interpreter runs is expected to match the MIME type: by default, an HTML document.
# The W3C Validator checks for valid markup in an HTML document. Everywhere that PHP generates an HTML document, that HTML document should be valid.

== Installing PHP ==

{{RequiredInstructions|content=

To install PHP 7.2, run the following command to install php7.2 from the Amazon Linux Extras repository.
sudo amazon-linux-extras install -y php7.2
sudo service php-fpm start

Once you install the required packages, [[Web Server Configuration#Restarting Apache and Testing|restart Apache]] for the changes to take effect.

}}

== PHP Configurations ==

{{RequiredInstructions|content=

The PHP configuration file is called ''php.ini''. In RHEL, it is located at ''/etc/php.ini''.

The configuration file is largely self-documenting as to what the different options mean. There is one value in particular that we want to change for the purposes of CSE330. Open php.ini in your favorite text editor, find the '''display_errors''' option, and set it to '''On''':

display_errors = On

You will need to restart Apache for any php.ini changes to take effect.

'''Note:''' You can turn errors on or off for a certain PHP script by using the '''error_reporting()''' and/or '''ini_set()''' functions.

'''Note:''' If you tell PHP to log its errors, they will be stored in the Apache error log file, at ''/var/log/php-fpm/www-error.log''.

}}

== Your First PHP Script ==

By default, all files that have the *.php extension will be interpreted as containing PHP code.
All files with PHP need to be saved as '.php'; not '.html'.
Here is the most basic PHP script:

<source lang="php">
<?php
phpinfo();
?>
</source>

Save that code into a file named phpinfo.php, and save it in your web server directory (/var/www/html/). Load it up in your browser like this: ''http://ec2-blah-blah.compute-1.amazonaws.com/phpinfo.php''

If you see something like this, your PHP installation is working!

[[Image:Php_info.png]]

Look at the source code of that page. Notice that even though there were only three lines of code in our PHP file, PHP filled it up with hundreds of lines of HTML!

Your own PHP scripts will probably look something more like this:

<source lang="php">
<!DOCTYPE html>
<html>
<head>
<title>My First PHP Script</title>
</head>
<body>
<?php
echo "\t<p>Hello World!</p>\n";
?>
</body>
</html>
</source>

The above code will send the following HTML to the browser:

<source lang="html4strict">
<!DOCTYPE html>
<html>
<head>
<title>My First PHP Script</title>
</head>
<body>
<p>Hello World!</p>
</body>
</html>
</source>

Here's the take-home message of this section. To run PHP code, enclose it in <code><?php ?></code> tags; HTML code can be used in a PHP file (although not inside the PHP code block!); and just a few lines of code can generate an arbitrary amount of HTML.

== PHP Language Components ==

PHP is a programming language much like others you have learned: it has variables, arrays, functions, objects etc. However, the syntax for PHP and the way PHP handles variables may be new to you. This section gives an overview of PHP the language.

=== Variables ===

All variables start with the currency '''$''' symbol. The '''$''' is necessary for both setting and accessing.

PHP is a weak-typed language; that is, you can set any type of value to a variable (string, array, number, etc), and PHP will accept it, even if that variable previously contained a different data type. In some ways this is convenient, but in others it can lead to hard-to-detect bugs, so be careful.

<source lang="php">
<?php
$i = 5; # Set the integer 5 to the variable $i
$msg = "Hello World"; # Set the string Hello World to the variable $msg
?>
</source>

Notice that all lines in PHP necessarily end with a semicolon. If you forget a semicolon, the PHP will fail to parse!

=== Strings ===

Strings in PHP can be either single-quoted or double-quoted. Single-quoted strings are always taken literally; double-quoted strings allow for additional elements.

<source lang="php">
<?php
$a = "Apple";
$b = "Banana";

$fruits_double = "$a\n$b"; # $fruits_double now contains the string Apple (line break) Banana
$fruits_single = '$a\n$b'; # $fruits_single now contains the string $a\n$b
?>
</source>

If you want to use special characters in a double-quoted strings, escape them using the backslash operator:

<source lang="php">
<?php
$a = "Apple";
$b = "Banana";

$fruits = "\$a\\n$b"; # $fruits now contains the string $a\nBanana
?>
</source>

You can concatenate multiple strings together using the concatenation operator, which in php is a period:

<source lang="php">
<?php
$c = "Cherry";
$g = "Grapefruit";

$fruits = $c."\n".$g; # $fruits contains the string Cherry (line break) Grapefruit
?>
</source>

=== Arrays ===

Declaring arrays is very simple in PHP.

<source lang="php">
<?php

$fruits = array("Apple", "Banana", "Cherry", "Grapefruit"); # $arr now contains an array with 4 items

echo $fruits[1]; # displays Banana

echo count($fruits); # displays 4

array_shift($fruits); # remove the first element of the array

echo $fruits[1]; # displays Cherry

echo count($fruits); # displays 3

?>
</source>

Note: '''echo''' outputs a string to the browser. We will see more ways to output data [[#Outputting Data|later in this guide]].

PHP also supports ''associative arrays''. Associative arrays are arrays whose keys are strings rather than numbers; in some languages, they are called Dictionaries.

<source lang="php">
<?php

$fruits["a"] = "Apple";
$fruits["b"] = "Banana";
$fruits["c"] = "Cherry";
$fruits["g"] = "Grapefruit";

echo $fruits["c"]; # displays Cherry

$keys = array_keys($fruits); # get an array containing indexed values of the fruits' keys

echo $keys[3]; # displays g

?>
</source>

=== Conditional Statements ===

Conditional statements work similarly to how they do in other languages.

<source lang="php">
<?php
$grade = 85;

if ($grade < 60) {
echo "F";
} elseif ($grade < 70) {
echo "D";
} elseif ($grade < 80) {
echo "C";
} elseif ($grade < 90) {
echo "B";
} else {
echo "A";
}
?>
</source>

PHP also supports switch statements, which are convenient for simplifying long if/elseif/else blocks:

<source lang="php">
<?php
$food = "Apple";

switch ($food) {

case "Apple":
case "Banana":
case "Cherry":
case "Grapefruit":
echo "You must like fruit!";
break;
case "Broccoli":
case "Spinach":
echo "You must like vegetables!";
break;
default:
echo "You don't like fruits or vegetables!";
break;

}
?>
</source>

=== Loops ===

PHP supports several different kinds of loops, including while, for, and foreach:

<source lang="php">
<?php

$mystring = "";
$i = 0;
while ($i < 5) {
$mystring .= "Blah";
$i++;
}
echo $mystring; # displays BlahBlahBlahBlahBlah

$myarray = array("Apple", "Banana", "Cherry", "Grapefruit");

# The following two loops are functionally equivalent.

$mystring = "";
for($i=0; $i<count($myarray); $i++){
$mystring .= $i.": ".$myarray[$i].", ";
}
echo $mystring; # displays 0: Apple, 1: Banana, 2: Cherry, 3: Grapefruit

$mystring = "";
foreach($myarray as $key => $value){
$mystring .= $key.": ".$value;
}
echo $mystring; #displays 0: Apple, 1: Banana, 2: Cherry, 3: Grapefruit

?>
</source>

=== Functions ===

Just like with all modern programming languages, you can define your own functions with PHP.

<source lang="php">
<?php

function add($x, $y){
return $x + $y;
}

echo add(3, 5); # displays 8

?>
</source>

A neat shortcut for PHP is that you can specify default parameter values directly in the argument list:

<source lang="php">
<?php

function sayHello($name="John Doe"){
return "Hello, $name... how do you do?";
}

echo sayHello("Todd"); # displays Hello, Todd... how do you do?
echo sayHello(); # displays Hello, John Doe... how do you do?

?>
</source>

=== Object-Oriented Programming ===

PHP5 supports object-oriented programming, and many new libraries come as classes rather than simple functions. Here is an example class definition:

<source lang="php">
<?php

class Food {
protected $name;

function __construct($name){
$this->name = $name;
}

static function getDefinition(){
return "Food is nourishment for carbon-based lifeforms.";
}

function formatName(){
return "Gotta love to eat " . $this->name;
}
}

class Fruit extends Food {
function formatName(){
return Food::formatName() . " (fruit)";
}
}

$fruit = new Fruit("Cherry");

echo $fruit->formatName() . "\n"; # displays: Gotta love to eat Cherry (fruit)

echo Fruit::getDefinition() . "\n"; # displays: Food is nourishment for carbon-based lifeforms.

?>
</source>

Things to notice:

* All methods in a class must be preceded by the function keyword.
* Static methods require the '''static''' keyword
* Instance methods can refer to the current instance through the implicit variable '''$this'''
* To access properties and methods from an instance, use an arrow: '''$instance->property'''
* To access static properties and methods from a class, use two colons: '''ClassName::StaticPropertyOrMethod'''
* Instances of classes are defined using the syntax '''$instance = new ClassName();''' like in Java

== Outputting Data ==

What's the point of a web language if we can't output data? The most basic method for outputting data is using either the '''echo''' or '''print''' statement:

<source lang="php">
<?php
echo "Hello World"; # outputs Hello World to the browser
?>
</source>

You often find yourself wanting to print all information associated with a variable for debugging purposes. PHP provides the handy '''var_dump''' function for that:

<source lang="php">
<?php
var_dump($myvar);
?>
</source>

=== Content Type ===

PHP is just a factory for generating documents. We've so far seen it create HTML documents, but it can just as easily create images, PDFs, or any other type of interpreted file format. Each of these files formats has a '''MIME Type''' associated with it. To specify a MIME type for a page served over HTTP, the <code>Content-Type</code> header must be set.

This brings up an important point. '''Every PHP document has an implicit <code>Content-Type: text/html</code> header.''' This means that by default, PHP will generate HTML documents. '''The MIME type for a document can be changed using the header() function:'''

<source lang="php">
<?php
header("Content-Type: text/plain");
echo time();
?>
</source>

The above example creates a plain text file containing the current timestamp (number of seconds since January 1, 1970).

For another application of the Content-Type header, see [[#Sending a File to the Browser]]

=== sprintf() ===

Code in PHP that outputs information can get very ugly. Consider, for example, the following snippet:

<source lang="php">
<?php
$medals = array(
"United States" => array(46, 29, 29),
"China" => array(38, 27, 23),
"Russia" => array(24, 26, 32)
);
?>

<table>
<tr>
<th>Country</th>
<th>Gold Medals</th>
<th>Silver Medals</th>
<th>Bronze Medals</th>
</tr>
<?php
foreach($medals as $country => $counts){
echo "\t<tr>\n\t\t<td>".htmlentities($country)."</td>\n\t\t<td>".htmlentities($counts[0])."</td>\n\t\t<td>".htmlentities($counts[1])."</td>\n\t\t<td>".htmlentities($counts[2])."</td>\n\t</tr>\n";
}
?>
</table>
</source>

This code will generate the following HTML:

<source lang="html4strict">
<table>
<tr>
<th>Country</th>
<th>Gold Medals</th>
<th>Silver Medals</th>
<th>Bronze Medals</th>
</tr>
<tr>
<td>United States</td>
<td>46</td>
<td>29</td>
<td>29</td>
</tr>
<tr>
<td>China</td>
<td>38</td>
<td>27</td>
<td>23</td>
</tr>
<tr>
<td>Russia</td>
<td>24</td>
<td>26</td>
<td>32</td>
</tr>
</table>
</source>

It's not very clear, though, that this is what the PHP code is going to generate. In particular, the ''echo'' line inside the ''foreach'' loop is 196 jumbled characters!

'''sprintf()''' is a nifty PHP function (with roots in C) that allows us to clean up that code. We get the same output as above if we use the following PHP instead:

<source lang="php">
<?php
$medals = array(
"United States" => array(46, 29, 29),
"China" => array(38, 27, 23),
"Russia" => array(24, 26, 32)
);
?>

<table>
<tr>
<th>Country</th>
<th>Gold Medals</th>
<th>Silver Medals</th>
<th>Bronze Medals</th>
</tr>
<?php
foreach($medals as $country => $counts){
echo sprintf("\t<tr>\n\t\t<td>%s</td>\n\t\t<td>%d</td>\n\t\t<td>%d</td>\n\t\t<td>%d</td>\n\t</tr>\n",
htmlentities($country),
$counts[0],
$counts[1],
$counts[2]
);
}
?>
</table>
</source>

The above code is more straightforward and self-explanatory than the first example, and it is also easier to edit and maintain.

Here's how '''sprintf()''' works. The first argument is a ''format string''. A format string can contain ''conversion specifications'' for additional data that will be inserted into the format string. The most common conversion specifications are:

* '''%s''' for a string
* '''%d''' for a signed integer
* '''%u''' for an unsigned integer
* '''%f''' for a floating-point number
** '''%.2f''' for a floating-point number with 2 decimal places
* '''%%''' for a literal percentage sign

For more detail on sprintf formats, refer to [http://us.php.net/sprintf the PHP documentation].

The remaining arguments are the data that are inserted into each of the conversion specifications. In the example above, there are four conversion specifications ('''%s''', '''%d''', '''%d''', and '''%d'''), so '''sprintf()''' takes 4 additional arguments after the format string.

Note that we still need to sanitize string output (using '''htmlentities()''' ) when we are using '''sprintf()'''. However, since our format string specifies integers for the remaining three arguments, '''sprintf()''' will take care of type casting, so we no longer need to worry about sanitizing the integers.

An additional tip: '''printf()''' is a shortcut for '''echo sprintf()'''. That is, the following lines are functionally equivalent:

<source lang="php">
<?php
$n = 12345.678;
printf("%.2f\n", $n); // prints 12345.68
echo sprintf("%.2f\n", $n); // also prints 12345.68
?>
</source>

== Passing Variables ==

Input values can be passed to a script either through the URL request line or through data submitted by a form. Built in arrays $_GET['var'] and $_POST['var'] return the value of ''var'' depending on the access method. As we will see in the next section, values can also be passed through session variables. Finally, you can use cookies to store and retrieve values.

Note that upon submitting a form, the page specified in the form tag's action attribute is loaded (with variables sent via post or get).

=== GET: Passing Variables via URL ===

If you want to pass values in a URL, the format is <nowiki>''http://example.com/yourphpfile.php?var1=value1&var2=value2.....''</nowiki>

PHP saves URL variables in the associative array '''$_GET'''.

For example, the following PHP document would print a hello message according to a name given in the URL.

<source lang="php">
<!DOCTYPE html>
<head><title>Hello World</title></head>
<body>
<?php
$name = $_GET['name'];

printf("<p>Hello, %s; how do you do?</p>\n",
htmlentities($name)
);
?>
</body>
</html>
</source>

If you access the URL like this: ''greeting.php?name=Alice'', it will show ''Hello, Alice; how do you do?''

'''Note:''' Before we output the name, we pass it through '''htmlentities()''', which sanitizes it for output to the HTML document. This is necessary because if we displayed the name verbatim from the URL, the name could be ''anything'': that includes malicious code that could be run on a victim's computer! You will learn more about this type of cyberattack, called '''Cross-Site Scripting''', in Module 4. For now, just know that whenever you display anything to the end user, you need to pass it through '''htmlentities()'''.

=== POST: Passing Variables via Form ===

Another method for passing variables from page to page is by sending it via POST variables. Like with GET, POST variables are saved in an associative array, this time named '''$_POST'''.

Consider the following form:

<source lang="html4strict">
<!DOCTYPE html>
<head><title>Hello World</title></head>
<body>
<form action="info.php" method="POST">
<p>
<label for="firstnameinput">First Name:</label>
<input type="text" name="firstname" id="firstnameinput" />
</p>
<p>
<label for="lastnameinput">Last Name:</label>
<input type="text" name="lastname" id="lastnameinput" />
</p>
<p>
<label for="birthyearinput">Birth Year:</label>
<input type="number" name="birthyear" id="birthyearinput" />
</p>
<p>
<strong>Gender:</strong>
<input type="radio" name="gender" value="male" id="maleinput" /> <label for="maleinput">Male</label>  
<input type="radio" name="gender" value="female" id="femaleinput" /> <label for="femaleinput">Female</label>
</p>
<p>
<input type="submit" value="Send" />
<input type="reset" />
</p>
</form>
</body>
</html>
</source>

For more information on the HTML markup in the above example, refer to [[HTML and CSS|the HTML and CSS guide]].

This submits a form with POST data to '''info.php'''. Inside '''info.php''', the provided values will be available in the associative $_POST array under the keys ''firstname'', ''lastname'', ''birthyear'', and ''gender''.

To display the submitted input on a target page, '''info.php''' could contain:

<source lang="php">
<!DOCTYPE html>
<head><title>Person Information</title></head>
<body>
<?php
$first = $_POST['firstname'];
$last = $_POST['lastname'];
$birthyear = (int) $_POST['birthyear'];
$gender = $_POST['gender'];
$age = 2012 - $birthyear;

printf("<p>Hello, %s %s! You were born in %d, so you are %d years old. You are %s.</p>\n",
htmlentities($first),
htmlentities($last),
$birthyear,
$age,
htmlentities($gender)
);
?>
</body>
</html>
</source>

'''Note:''' It is ''very'' important that you always sanitize strings that you output to the browser using '''htmlentities()''' or a similar function. If you don't, your site will be vulnerable to all sorts of malware. For more information, refer to the [[Web Application Security, Part 1|Web Application Security]] guide.

For further help with using HTML forms with PHP, we recommend [http://www.tizag.com/phpT/forms.php this tutorial].

=== Self-Submitting Forms ===

Oftentimes we don't want to pass form values to a different page, but rather to the same page as contains the form so that we can process the information and display output on that same page. To do this, you can simply omit the '''action''' parameter from the form. Alternatively, you can use a PHP environment variable like $_SERVER['PHP_SELF'].

The following example simply takes the data input from a form and, via POST variables, echoes it back to the user as bold text on the same page:

<source lang="php">
<!DOCTYPE html>
<html>
<head><title>Bold Printer</title></head>
<body>
<form action="<?php echo htmlentities($_SERVER['PHP_SELF']); ?>" method="POST">
<p>
<label for="name">Name:</label>
<input type="text" name="name" id="name" />
</p>
<p>
<input type="submit" value="Print in Bold" />
</p>
</form>

<?php
if(isset($_POST['name'])){
printf("<p><strong>%s</strong></p>\n",
htmlentities($_POST['name'])
);
}
?>
</body>
</html>
</source>

== Sessions ==

If you need to keep track of information associated with a user between page loads, you should use a ''session''. Sessions are excellent for keeping track of the currently-logged-in user, for example.

PHP makes session management dirt simple. Whenever you need to use session variables, simply call '''session_start()'''. Then, you may interact with the $_SESSION associative array, and PHP will take care of accessing and saving all session variables for you!

For example, the following page will display a number, and then double that number on each page reload:

<source lang="php">
<?php
session_start();

$oldnum = (int) $_SESSION['inc_num'];
if($oldnum<1) $oldnum=1;
$newnum = $oldnum*2;

echo $newnum;
$_SESSION['inc_num'] = $newnum;
?>
</source>

Sessions also work across different pages, as long as they are on the same web server.

'''Tip:''' To log out a user, you can call '''session_destroy()''' after '''session_start()''' on the logout page.

== Other PHP Tips ==

=== Redirecting to a Different Page ===

If you want to redirect the user to a different page, you need to send a '''Location''' header. For example, a page containing only this code would redirect the user to the ''login.php'' file:

<source lang="php">
<?php
header("Location: login.php");
exit; // we call exit here so that the script will stop executing before the connection is broken
?>
</source>

It is often useful to perform redirects after you perform server-side operations.

'''Important:''' You should generally not leave a user sitting on a page that has POST data, because if they refresh the page, their POST data will be submitted again, causing your server to perform an action twice. Rather, you should perform the operation, and then redirect them to another page that does not contain POST data in the header.

=== Reading a File Line-by-Line ===

If you want to read a file line-by-line, open a file handler using '''fopen()''', use '''fgets()''' to read lines until you've reached the end, and then close the file using '''fclose()'''.

The following example reads '''file_with_lines.txt''' and outputs its content in an unordered list, where each list item is a line in the file:

<source lang="php">
<?php
$h = fopen("file_with_lines.txt", "r");

$linenum = 1;
echo "<ul>\n";
while( !feof($h) ){
printf("\t<li>Line %d: %s</li>\n",
$linenum++,
fgets($h)
);
}
echo "</ul>\n";

fclose($h);
?>
</source>

'''Note:''' Most files in Linux end with an empty line, so the very last iteration of the above loop will output an empty bullet point.

'''Note:''' The function '''fgets()''' reads the contents of a line ''in addition to the line feed at the end of the line''. If you want to compare a string to the line that was read by '''fgets()''', you need to pass the value returned by '''fgets()''' to a function like '''[http://php.net/trim trim()]'''.

'''Note:''' If you want to read an entire file, PHP provides the '''file_get_contents()''' function, which is basically a shortcut for '''fopen()''', '''fread()''', and '''fclose()'''. ('''fread()''' is similar to '''fgets()''', except it reads the entire file instead of just one line.)

=== Sending a File to the Browser ===

Another useful function to know is the '''readfile''' function. It takes a file on the webserver's file system and sends it to the web browser. A simple example is given below, but note that you'll probably need to specify addition header fields in order to have the web browser download the file correctly.

The following example loads a file associated with a certain username and displays it.

<source lang="php">
<?php
session_start();

$filename = $_GET['file'];

// We need to make sure that the filename is in a valid format; if it's not, display an error and leave the script.
// To perform the check, we will use a regular expression.
if( !preg_match('/^[\w_\.\-]+$/', $filename) ){
echo "Invalid filename";
exit;
}

// Get the username and make sure that it is alphanumeric with limited other characters.
// You shouldn't allow usernames with unusual characters anyway, but it's always best to perform a sanity check
// since we will be concatenating the string to load files from the filesystem.
$username = $_SESSION['username'];
if( !preg_match('/^[\w_\-]+$/', $username) ){
echo "Invalid username";
exit;
}

$full_path = sprintf("/srv/uploads/%s/%s", $username, $filename);

// Now we need to get the MIME type (e.g., image/jpeg). PHP provides a neat little interface to do this called finfo.
$finfo = new finfo(FILEINFO_MIME_TYPE);
$mime = $finfo->file($full_path);

// Finally, set the Content-Type header to the MIME type of the file, and display the file.
header("Content-Type: ".$mime);
header('content-disposition: inline; filename="'.$filename.'";');
readfile($full_path);

?>
</source>

=== Uploading a File ===

Uploading files is similar to submitting data in forms. For starters, you need a form that submits to your uploading script:

<source lang="html4strict">
<form enctype="multipart/form-data" action="uploader.php" method="POST">
<p>
<input type="hidden" name="MAX_FILE_SIZE" value="20000000" />
<label for="uploadfile_input">Choose a file to upload:</label> <input name="uploadedfile" type="file" id="uploadfile_input" />
</p>
<p>
<input type="submit" value="Upload File" />
</p>
</form>
</source>

Note that we need to specify <code>enctype="multipart/form-data"</code> on the form. This is to enable files to be uploaded and recognized by PHP.

PHP puts all files submitted with a form into the '''$_FILES''' superglobal array. Each entry in the $_FILES array contains the following information:

* '''$_FILES['uploadedfile']['name']''' is the original name of the file on the client's machine
* '''$_FILES['uploadedfile']['type']''' is the MIME type of the file according to the browser (don't take it for granted)
* '''$_FILES['uploadedfile']['size']''' is the size, in bytes, of the uploaded file
* '''$_FILES['uploadedfile']['tmp_name']''' is the temporary location on your filesystem where Apache saved the uploaded file
* '''$_FILES['uploadedfile']['error']''' is the error code associated with the file upload (if applicable)

To upload the file submitted by the form above in a user's special folder, we could use the following PHP code:

<source lang="php">
<?php
session_start();

// Get the filename and make sure it is valid
$filename = basename($_FILES['uploadedfile']['name']);
if( !preg_match('/^[\w_\.\-]+$/', $filename) ){
echo "Invalid filename";
exit;
}

// Get the username and make sure it is valid
$username = $_SESSION['username'];
if( !preg_match('/^[\w_\-]+$/', $username) ){
echo "Invalid username";
exit;
}

$full_path = sprintf("/srv/uploads/%s/%s", $username, $filename);

if( move_uploaded_file($_FILES['uploadedfile']['tmp_name'], $full_path) ){
header("Location: upload_success.html");
exit;
}else{
header("Location: upload_failure.html");
exit;
}

?>
</source>

'''Note:''' A lot of the code above is the same as the code in the file download example. In your project, you should save the logic of generating the file path (in a function) in a third file that you include into the upload and download scripts.

'''Note:''' By default, PHP will not allow a file upload of more than 2 MB. SitePoint has an excellent tutorial for how to increase this limit: http://www.sitepoint.com/upload-large-files-in-php/

== PEAR ==

The PHP Extension and Application Repository, or '''PEAR''', is a package manager that enables you to install classes and/or extensions to PHP. In other words, it is a yum/apt for PHP. You can think of PEAR as PHP's analog of ''gem'' in Ruby or ''pip'' in Python.

=== Configuration ===

PEAR is available through both the yum and apt repositories under the package name '''php-pear'''.

<source lang="bash">
$ sudo yum install php-pear # in RHEL
</source>

=== Installing PHP Extensions ===

Using PEAR to install PHP extensions is just as easy as yum, apt, gem, and pip. Just run '''sudo pear install xxx'''.

<source lang="bash">
$ sudo pear install Archive_Tar
</source>

Go here for a list of available packages in PEAR: http://pear.php.net/packages.php

=== Using PHP Extensions ===

To make use an extension installed in PEAR, you must '''include''' or '''require''' it into the PHP file of interest. For example, if you wanted to use the Archive_Tar class from the above example in a PHP file, you would do:

<source lang="php">
<?php
require "Archive/Tar.php";

// your code here
?>
</source>

By convention, the filepath you need to specify is the name of the PEAR package with underscores replaced with slashes and a .php extension at the end. For example, the package ''Console_Getargs'' becomes <code>require "Console/Getargs.php";</code>

=== PECL ===

The PHP Extension Community Library, or '''PECL''', is a repository for additional PHP extensions, the difference being that PECL extensions are written in compiled C code rather than PHP.

PECL is installed automatically when you install PEAR from yum or apt-get.

Installing an extension using PECL is almost identical to PEAR, except that you use '''pecl''' instead of '''pear''' on the command prompt:

<source lang="bash">
$ sudo pecl install imagick
</source>

You do not need to '''require''' or '''include''' PECL extensions since they are already compiled into PHP.

== PHP Reference ==

If you ever have a question about PHP, first check the well-written documentation. To skip immediately to the documentation on any function, simply visit <nowiki>http://php.net/</nowiki>'''functionname'''. For example: http://php.net/var_dump

[[Category:Module 2]]

VS Code

2023-01-27T20:08:41Z

Zach: /* SFTP Integration (Shipping files to your server) */

== Getting Started With Each Module ==

* Create Github repo using the CSE 330 repo creator (Check Piazza for the link)
* Navigate to the created repo in Github
* Clone the repo via https to a dedicated folder on your computer

[[File:File-Structure.png|400px|center]]

* In VS Code, File > Open the Spring2019-module1-individual-STUDENTID '''folder''' that you cloned to your computer

If you followed the above steps correctly, the folder's Git should be configured to your Github repository! I highly recommend creating your Github repo, cloning the folder to a designated spot, and keeping all modules separated for the semester as shown in the image above.

== Module 1 -- Using the Git Integration ==
'''Note: If you're using Windows, you may need to install Git on your machine first. https://git-scm.com/download/win . You can check if you have Git running by typing git --version in your terminal'''

* First, open the File Explorer tab on the left, and right click to create a new file, birthday.html
[[File:New-file.png|400px|center]]
* Ensure VS Code has it marked as an html file (see the file type next to the smiley face)
* Type '''doc''' followed by enter, and allow VS Code to generate a basic HTML file for you (use VS Code to your advantage! It can do a lot of your coding for you).
* Save the file, you should see a popup next to the Version Control symbol on the left. VS Code has detected that your project is different than what it is in your remote Github repo.
* To commit, simply type a message in the box and click the check mark above it. To push to your Github repo, click the 3 dots next to the check mark and select 'push'
* Note this is also where you can pull from your remote.

==== Bonus - Module 1 ====
* If you like your text editor to look pretty, you can go to the Extensions tab and search for different themes. (Check out the Material theme!)
* You can view your settings by clicking CMD + , (Mac) or Ctrl + , (Windows). Search for Theme and choose one that works for you!

== Module 2 ==
=== DISCLAIMER ===
{{RequiredInstructions|content=

<span style="color:red">There are two SFTP VSCode extensions of the same name. One of them is depreciated. Use the extension distributed by Natizyskunk.</span>

}}

=== SFTP Integration (Shipping files to your server) ===
# Select a folder to use for your work and open it in VSCode. When editing files on your instance, the file will be saved locally in this location for editing and reuploaded
# Open the Extensions tab and search for 'SFTP'. Download the version distributed by "Natizyskunk".
# To use commands in VS Code, type '''Cmd + Shift + P''' (Mac), or '''Ctrl + Shift + P''' on windows.
# Type SFTP and open the sftp config command.

Your sftp.json file should look like the following:
{
"name": "<My Instance>",
"host": "<ec2-xxxxx-xxxxxx-xxxxx.amazonaws.com>",
"protocol": "sftp",
"port": 22,
"username": "<custom instance username>"
"remotePath: "/home/<custom instance username>/",
"uploadOnSave": true,
"privateKeyPath": "</path/to/.ssh/id_rsa>"
}
- Note the uploadOnSave field is optional, but can be nice!
- Check the VS Code output for errors in your config.

Additionally: You should notice an SFTP tab on the side of VS Code. Here, you can view the file structure on your server.]

'''Preventing "Edit in Local" when editing files'''

In order to edit locally, you'll have to right click on the file you wish to edit and select "Edit in local" every time. To avoid this, you can:
# Open vscode settings by File >> Preferences >> Settings
# In settings, search for "downloadWhenOpenInRemoteExplorer" and toggle it on
# restart VScode.

=== Testing Everything Locally ===
If you would like to avoid having to frequently upload your changes to your instance in order to test your changes, you can optionally set up a local PHP server.

'''To run PHP locally'''

'''Windows'''
# Download WAMP http://www.wampserver.com/en/download-wampserver-64bits/
# Place your project repositories in Wamp's www/ directory, and navigate to http://localhost in your browser. You should see your projects listed here, run php files, and use phpmyadmin.

'''Mac'''
# Download Homebrew. Homebrew is a great package manager for OSx and we will be using it in future modules.
# Type the following in your terminal and follow instructions
/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
# In your terminal, type '''php -v'''. If your version is less than 7, update with brew. '''brew install php'''
# In VS Code, search for the extension '''Php Server'''. After installing, use VS Code's command pallette with CMD+Shift+P , search for PHP, and select start server. You should be able to view your project in your localhost

Module 4

2023-01-10T19:34:25Z

Zach:

__NOTOC__
In Module 4, you will learn the fundamentals of Python, a popular scripting language, and Regular Expressions, which give you power in text processing.

This article contains your assignments for Module 4.

== Using the Wiki ==
{{RequiredInstructions|content=
Text enclosed by <syntaxhighlight lang="bash" inline><</syntaxhighlight> and <syntaxhighlight lang="bash" inline>></syntaxhighlight> should be replaced by content unique to you.

'''Example'''

<source lang="bash">
$ sudo useradd -r -m -c "<My Full Name>" <usernameHere>
</source>
becomes
<source lang="bash">
$ sudo useradd -r -m -c "Zach Cohn" zcohn
</source>
}}

== Reading ==

The following articles on the online class wiki textbook contain information that will help you complete the assignments.

* [[Python]]
* [[Regular Expressions]]
* [[FAQ - Mod 4]]

== Individual Assignments ==

=== Learn Python and Regular Expressions ===

Up to this point in CSE 330, you have used only one programming language: PHP. In Module 4, you will learn a new language: Python. Before starting this assignment, you should look over the [[Python|online textbook article about Python]]. You should be using version 3 (or greater) of Python. Additionally, you may find it helpful to look over the [[Regular Expressions|online textbook article about Regular Expressions]].

=== Write some Regular Expressions ===

Please write regular expressions that do each the following:

# Match the substring "hello world" in a string.
# Find all character strings in an input string that contains three or more consecutive vowels, regardless of case.
# Match an input string that is '''entirely''' a flight code, of the format AA####, where '''AA''' is a two-letter uppercase airline code, and '''####''' is a three- or four-digit flight number.
Examples:
Note: text in ''[[red]]'' is matched by the regular expression. Please note that each line is a '''separate''' test input string. For Regex 3, '''six''' separate test cases are given.
Regex 1: Programmers will often write ''[[hello world]]'' as their first project with a programming language.
Regex 2: The ''[[gooey]]'' peanut butter and jelly sandwich was a ''[[beauty]]''.
Regex 3: ''[[AA312]]''
''[[AA1298]]''
''[[NW1234]]''
''[[US443]]''
US31344
AA123 extratext

Save your regexes to text files – one each.

=== Baseball Stats Counter ===

The St. Louis Cardinals are the most legendary baseball team in the national league. In this exercise, you will be creating a Python script that reads box scores from a file and computes the Cardinals' players' batting averages in a particular season.

==== Tips and Instructions ====

* You should write a Python script file to solve this problem.
* You should use a regular expression to parse players' names, at-bats, hits, and runs from the input file.
* You may want to create a class to hold and compute information about each player.
* Your file should take one command-line argument: the path to an input file. If no path is given, your program should print a usage message.

==== Input ====

Sample input files may be found here: http://classes.engineering.wustl.edu/cse330/content/cardinals/

The top of the file contains a citation, and what follows is an entire season of Cardinals Baseball box scores. Each game starts with a title; for example:

=== St.Louis Cardinals vs. Chicago Cubs, 1940-04-19 ===

What follows are the performance of each player in that game. The format of each line is as follows:

XXX batted # times with # hits and # runs

where '''XXX''' is the player's name (you may assume that the name is unique), and each '''#''' are integers representing the number of "at bats", the number of "hits", and the number of "runs", respectively.

==== Output ====

Your script needs to output all players' batting average across the season. A player's batting average is that player's total hits divided by that player's total at-bats throughout the entire season. For example, Johnny Hopp was at bat 149 times in 1940, and he had 41 hits, so his batting average was 0.275.

Each line in the output should have the format:

XXX: #.###

where '''XXX''' is the player's name, and '''#.###''' is the player's batting average, '''rounded''' (not truncated) to three decimal places.

The players should be sorted by batting average, with the highest batting average on top. If two players have the exact same batting average (before rounding occurs), the order between those two players is unimportant.

For example, the correct output for the 1940 season is:

<source lang="text">
Pepper Martin: 0.316
Walker Cooper: 0.316
Johnny Mize: 0.314
Ernie Koy: 0.310
Enos Slaughter: 0.306
Joe Medwick: 0.304
Terry Moore: 0.304
Joe Orengo: 0.286
Jimmy Brown: 0.280
Marty Marion: 0.279
Don Gutteridge: 0.276
Johnny Hopp: 0.275
Creepy Crespi: 0.273
Mickey Owen: 0.265
Bill DeLancey: 0.250
Don Padgett: 0.242
Stu Martin: 0.238
Eddie Lake: 0.222
Hal Epps: 0.214
Lon Warneke: 0.209
Harry Walker: 0.185
Max Lanier: 0.179
Bill McGee: 0.178
Carl Doyle: 0.174
Mort Cooper: 0.157
Clyde Shoun: 0.145
Carden Gillenwater: 0.130
Bob Bowman: 0.067
</source>

== Grading ==

We will be grading the following aspects of your work. There are 50 points total.

'''Assignments (including code) must be committed to Github by the end of class on the due date (commit early and often). Failing to commit by the end of class on the due date will result in a 0. '''

Upload your baseball code and a file with your Regular expressions.

# '''Regular Expressions (15 Points):'''
#* '''Your regex files should contain your regex (with no delimiters or flags) and nothing else.'''
#* Regular Expression 1 Correct and saved in a file named <code>regex1.txt</code> (5 points)
#* Regular Expression 2 Correct and saved in a file named <code>regex2.txt</code>(5 points)
#* Regular Expression 3 Correct and saved in a file named <code>regex3.txt</code>(5 points)
# '''Baseball Stats Counter (35 Points):'''
#* Solution is written entirely in Python and saved in a file named <code>baseball.py</code> (8 points)
#*: ''Note: Failing to write your code in Python 3 will result in losing, at a minimum, points for this category.''
#* Correct usage of one or more regular expressions to parse and extract data from each line of the input file (8 points)
#*: ''Note: You should not be using <code>str.split</code> to extract data.''
#* Script prints a usage message if a command line argument is not present (4 points)
#*: ''Note: For an example of a usage message, see [https://en.wikipedia.org/wiki/Usage_message#Examples this link.]''
#* Output is correct for all test cases (15 points)
#*: ''This includes sorting and rounding.''

[[Category:Module 4]]
[[Category:Modules]]

Module 3

2023-01-10T19:33:42Z

Zach:

__NOTOC__
In Module 3, you will learn about MySQL, a web application database.

This article contains your assignments for Module 3.

== Using the Wiki ==
{{RequiredInstructions|content=
Text enclosed by <syntaxhighlight lang="bash" inline><</syntaxhighlight> and <syntaxhighlight lang="bash" inline>></syntaxhighlight> should be replaced by content unique to you.

'''Example'''

<source lang="bash">
$ sudo useradd -r -m -c "<My Full Name>" <usernameHere>
</source>
becomes
<source lang="bash">
$ sudo useradd -r -m -c "Zach Cohn" zcohn
</source>
}}

== Reading ==

The following articles on the online class wiki textbook contain information that will help you complete the assignments.

* [[Introduction to MySQL]]
* [[MySQL Schema and State]]
* [[PHP and MySQL]]
* [[Web Application Security, Part 2]]
* [[FAQ - Mod 3]]

== Individual Assignments ==

You will create a simple grades database containing four tables.

'''IMPORTANT:''' The individual assignment is to be done by writing queries by hand. (You can use phpMyAdmin to run the queries, but '''do not let phpMyAdmin write your queries for you!''') Writing your own queries gives you a greater understanding of the underlying mechanisms and databases in general. Additionally, having SQL knowledge and experience is a very marketable trait.

=== Install MySQL and phpMyAdmin ===

Before you start, '''you need to install MySQL''' on your EC2 instance. You may also install phpMyAdmin, a web frontend to MySQL, if you choose to do so. Instructions to install both can be found on the [[Introduction to MySQL]] page.

=== Set Up a Database ===

You may find the following article to be very helpful: [[Introduction to MySQL#Managing Databases]]

# Create a database named '''wustl'''
# Create a user named '''wustl_inst''' and give them the password '''wustl_pass'''

=== Create Tables ===

You may find the following article to be very helpful: [[MySQL Schema and State]]

When creating tables, keep the following items in mind:

* You should create all tables such that they use the InnoDB storage engine, since we wish to make use of its support of foreign keys. In other database engines, foreign key constraints are not enforced. (If you mess up, you can use ALTER TABLE to change the storage engine.)
* Where it is appropriate, you should take advantage of the ability to define fields NOT NULL so that you do not inadvertently insert incomplete data for a row.

Create the following tables:

# Create a table named '''students''' with the following fields:
#* '''id''' of an appropriately-sized unsigned integer type (we will never need to process more than a million students)
#* '''first_name''' of type '''VARCHAR(50)'''
#* '''last_name''' of type '''VARCHAR(50)'''
#* '''email_address''' of type '''VARCHAR(50)'''
#: The primary key should be on the '''id''' field.
# Create a table named '''departments''' with the following fields:
#* '''school_code''' of type '''ENUM''' (e.g. "L" for ArtSci and "E" for Engineering). The options are as follows:
#*: 'L', 'B', 'A', 'F', 'E', 'T', 'I', 'W', 'S', 'U', 'M'
#* '''dept_id''' of an appropriately-sized unsigned integer type (in the foreseeable future there will never be more than 200 departments in a school)
#* '''abbreviation''' of type '''VARCHAR(9)''' (e.g. CSE, ChemE, etc)
#* '''dept_name''' of type '''VARCHAR(200)''' (e.g. Computer Science and Engineering)
#: The primary key should be on two fields: '''school_code''' and '''dept_id'''
#: '''Note:''' Why not just use the department ID for the primary key? The ID numbers are sometimes reused across schools. For instance, department 33 in The College of Arts and Sciences is Psychology while department 33 in The School of Engineering and Applied Sciences is Energy, Environmental and Chemical Engineering. Thus we must also include the school code to differentiate them such that the record for E33 is distinct from that of L33.
# Create a table named '''courses''' with the following fields:
#* '''school_code''' of type '''ENUM'''
#*: This should have the same letters in the same order as the '''school_code''' '''ENUM''' field in the '''departments''' table.
#* '''dept_id''' of the same size of integer as in the '''departments''' table
#* '''course_code''' of type '''CHAR(5)''' (this will hold course codes; e.g., '330S', '131', '2960')
#* '''name''' of type '''VARCHAR(150)'''
#: The appropriate fields in this table should make a foreign key reference to the corresponding fields in '''department'''.
#: In addition, it is up to you to determine the field(s) for the primary key.
#: '''Note:''' If you are unsure which fields to make the foreign keys and feel the need to beg your friends or the TAs to give you the answer, we recommend exercising reason and common sense.
# Create a table named '''grades''' with the following fields:
#* '''pk_grade_ID''' of an appropriately-sized unsigned integer type (should be larger than the one you chose for the '''students''' table). There will never be more than 1,000 grades per student.
#*: You should declare this field to be '''auto_increment''' and make it the primary key.
#*: This is called a ''surrogate key'' as opposed to a ''natural key'' because it doesn't arise "naturally" from the data; that is to say that it is not derived from application data. The use of surrogate keys vs. natural keys is a polemic issue amongst database designers. For a decently unbiased take on the issue, see this page: http://www.agiledata.org/essays/keys.html#Comparison
#*: '''Food for thought:''' Aside from simply exposing you to the concept, can you think of any reasons why a surrogate key might be preferable for this table? Alternatively, what attributes, if any, could you use as a natural key, and what might be the drawbacks of doing so?
#* '''student_id''' of the same integer (unsigned) type that you chose in the '''students''' table
#* '''grade''' of type '''decimal(5,2)'''
#* '''school_code''' of type '''ENUM'''
#*: Can you guess what entries to use for the enum?
#* '''dept_id''' of the same integer type that you've used for this field in other tables
#* '''course_code''' of the same type you chose for the course code in the '''courses''' table
#: The '''grades''' table should have foreign keys to both the '''students''' table and the '''courses''' table.

=== Populate Tables ===

Now, populate the tables by downloading the text files given below and loading them into the appropriate tables.

'''Note:''' Do not insert all the values manually; instead, use the MySQL's ability to load in data from text files.

You may find the following article to be very helpful: [[MySQL Schema and State]]

The files containing the data are:
* students: https://classes.engineering.wustl.edu/cse330/content/students_data.txt
* courses: https://classes.engineering.wustl.edu/cse330/content/courses.txt
* departments: https://classes.engineering.wustl.edu/cse330/content/departments_data.txt
* grades: https://classes.engineering.wustl.edu/cse330/content/grades_data.txt

=== Insert More Data ===

Insert the following data into the tables that you have created and populated:

# Insert entries for CSE 330S, CSE 131, and CSE 260M.
#: ''If you don't know CSE's department code, how can you find out using the tables in the database?''
# Insert the grades given for the students named below.
#* '''Ben Harper'''
#** E-mail: '''bharper@ffym.com'''
#** Student ID: '''88'''
#** Grades:
#*** 35.5 in CSE 330S
#*** 0 in CSE 131
#*** 95 in CSE 260M
#* '''Matt Freeman'''
#** E-mail: '''mfreeman@kickinbassist.net'''
#** Student ID: '''202'''
#** Grades:
#*** 100 in CSE 330S
#*** 90.5 in CSE 131
#*** 94.8 in CSE 260M
#* '''Marc Roberge'''
#** E-mail: '''mroberge@ofarevolution.us'''
#** Student ID: '''115'''
#** Grades:
#*** 75 in CSE 330S
#*** 37 in CSE 131
#*** 45.5 in CSE 260M

=== Querying Your Database ===

Now that you have a fully-functional, populated database, let's do some queries on it!

Please Running the following <code>SELECT</code> queries, then copy the output to a text file of the form <code>query1.txt</code>, <code>query2.txt</code>, and so on, and commit them with the rest of your repository (be sure to include your query command and the entire response, including the number of results found). For each query, only the requested columns should be present. You will receive 0 points for any query that includes additional columns.

# Select the entire grades table.
# Select all fields describing the courses offered in the College of Arts & Sciences (school code <code>L</code>).
# The names, student IDs, and CSE330 grades of all students who are in CSE330S.
#: '''Note:''' This query should involve joins. You don't need to use any aggregation functions.
# The names, e-mails, and average grades of any student with an average below 50 so that the dean can send them an email notification that they are now on academic probation.
#: ''You should be able to do this in only one query, without making any temporary tables. You will need to use aggregation functions and the <code>HAVING</code> keyword.''
# An individual report card for Jack Johnson, consisting of only his student ID, e-mail address, and average grade.
#: ''Again, you should be able to do this in just one query, using the correct combination of aggregation functions and joins.''
#: '''Note:''' Your query must look for students with the name "Jack Johnson" instead of hard-coding the student ID.

== Group Project ==

You will work in pairs on this project. You may work with the same partner from Module 2, or change to someone else.

'''Important Reminder:''' frequently commit your work to your Git repository as a backup!

=== Simple News Web Site ===

'''Examples:'''
* http://digg.com/
* http://slashdot.org/
* https://news.ycombinator.com/

You should use '''PHP''' unless you get permission to use another web technology from the instructor. You may also use phpMyAdmin to help set up databases.

You may find this wiki article helpful: [[PHP and MySQL]]

==== Requirements ====

* Users can register for accounts and then log in to the website.
* Accounts should have both a username and a ''secure'' password. '''NEVER store plaintext passwords in a database!'''
*: For more information on password security, refer to [[Web Application Security, Part 2#Password Security|the Web Application Security guide]].
* Registered users can submit story commentary.
*: A link can be associated with each story, and they should be stored in a separate database field from the story.
*: For example, a story that included a link could look like the following:
*:: Title: Check out this cool search engine I found!
*:: Body: This search engine that I found has been very helpful with all of my homework assignments. Searching my error messages helps me debug.
*:: Link: http://google.com
* Registered users can comment on any story.
* Unregistered users can only view stories and comments.
* Registered users can edit and delete their stories and comments.
* All data must be kept in a MySQL database (user information, stories, comments, and links).
* Creative Portion.

=== Web Security and Validation ===

No application is finished until much thought is put into web security and best practice. Throughout this course, we heavily emphasize the dogma of [https://mirrors.meiert.org/coderesponsibly.org/ responsible] coding.

Read this week's Web Application Security guide: [[Web Application Security, Part 2]]. In particular, your project needs to:

* '''Your application needs to be secure against SQL injection attacks'''. If you are using prepared queries, you should already be safe on this front.
* '''Your passwords need to be securely hashed salted''' to prevent rainbow table attacks.
* '''You should pass tokens in forms''' to prevent CSRF attacks.
* '''You should check all preconditions on the server side''' to prevent Abuse of Functionality attacks.

You shouldn't forget the practices you learned last week:

* '''Your page should validate''' with no errors or warnings through the W3C validator.
* '''Filter Input and Escape Output''', but not the other way around.

== Grading ==

We will be grading the following aspects of your work. There are 100 points total.

'''Assignments (including code) must be committed to Github by the end of class on the due date (commit early and often). Failing to commit by the end of class on the due date will result in a 0. '''

'''If you do not include a link to your group portion running on your instance in your group portion README.md, you will receive a 0 for the group portion.'''
________

# '''MySQL Queries (25 Points):'''
#* A MySQL server is running on your instance. Put a screenshot in your repository of your terminal connected to your MySQL server. ( (2 points)
#* Tables' fields, including data types and foreign keys, are correct (8 points)
#*: ''Note: To demonstrate the structure of your tables, you should commit a text file containing the output of the <code>SHOW CREATE TABLE</code> command '''for each table.''' Each table's file should have a name of the form <code>tablename.sql</code> (e.g. <code>students.sql</code>).''
#* The output of each of the five queries is correct. (3 points each)
#*: ''Note: To demonstrate the output of your queries, text files containing the 'entire' result (including the query and number of records found) of running each query should committed to Github. The file for query 1 should be named <code>query1.txt</code>, the file for query 2 should be named <code>query2.txt</code>, and so on.''
# '''News Site (60 Points):'''
#* '''''User Management (20 Points):'''''
#** A session is created when a user logs in (3 points)
#** New users can register (3 points)
#** Passwords are hashed, salted, and checked securely (3 points)
#**: ''Note: You will receive 0 points for this section if you use the <code>==</code> or <code>===</code> operators to compare password hashes, or if you use the <code>crypt</code> or <code>md5</code> functions at any point.''
#** Users can log out (3 points)
#** A user can edit and delete his/her own stories and comments but cannot edit or delete the stories or comments of another user (8 points)
#* '''''Story and Comment Management (20 Points):'''''
#** Relational database is configured with correct data types and foreign keys (4 points)
#*: ''Note: To demonstrate the structure of your database, you should commit a 'single' text file containing the output of the <code>SHOW CREATE TABLE</code> command '''for all tables in your database,''' called <code>tables.sql</code>. This one file should contain the output for all of your tables.
#** Stories can be posted (3 points)
#** A link can be associated with each story, and is stored in a separate database field from the story (3 points)
#** Comments can be posted in association with a story (4 points)
#** Stories can be edited and deleted (3 points)
#** Comments can be edited and deleted (3 points)
#**: ''Note: Although there are only 6 points allocated for editing/deleting in this section, there are 8 more points at stake in the User Management section that cannot be earned unless editing/deleting is implemented. Implementing editing but not deleting, or vice-versa, will result in earning half the points.''
#* '''''Best Practices (15 Points):'''''
#** Code is well formatted and easy to read, with proper commenting (3 points)
#** Safe from SQL Injection attacks (2 points)
#** Site follows the FIEO philosophy (3 points)
#** All pages pass the W3C HTML and CSS validators (2 points)
#** CSRF tokens are passed when creating, editing, and deleting comments and stories (5 points)
#* '''''Usability (5 Points):'''''
#** Site is intuitive to use and navigate (4 points)
#** Site is visually appealing (1 point)
# '''Creative Portion (15 Points)''' (see below)
#* '''''Make sure you have a README.md file in your group repo with the following:'''''
#** Names and Student IDs of all the group members
#** A link to your homepage of the site
#** A brief description of what you did for your creative portion
#** Any additional login details needed for the TA

[[Category:Module 3]]
[[Category:Modules]]

Module 2

2023-01-10T19:29:02Z

Zach:

__NOTOC__
Module 2 introduces you to Linux, a command-line environment, the Apache web server, and PHP. You will create and configure your own cloud instance, install Apache and related software, and then form groups to make a simple file sharing site.

<span style="font-size:1.3em; line-height:1.2em; color:blue;">'''DO NOT WAIT UNTIL THE LAST MINUTE TO START ON THIS (or any) MODULE!''' The most common reason students perform poorly in this class is procrastination. You have been warned.</span>

== Using the Wiki ==
{{RequiredInstructions|content=
Text enclosed by <syntaxhighlight lang="bash" inline><</syntaxhighlight> and <syntaxhighlight lang="bash" inline>></syntaxhighlight> should be replaced by content unique to you.

'''Example'''

<source lang="bash">
$ sudo useradd -r -m -c "<My Full Name>" <usernameHere>
</source>
becomes
<source lang="bash">
$ sudo useradd -r -m -c "Zach Cohn" zcohn
</source>
}}

== Reading ==

The following articles on the online class wiki textbook contain information that will help you complete the assignments.

* [[Linux]]
* [[Bash]]
* [[Amazon Web Services]]
* [[SSH]]
* [[Apache]]
* [[PHP]]
* [[Workflow]]
* [[VS Code]]
* [[Web Application Security, Part 1]]
* [[FAQ - Mod 2]]

== Individual Assignments ==

{{RequiredInstructions|content=

<span style="color:red">There are a lot of wiki pages that you need to read in order to complete the individual portion of Module 2. The wiki pages contain an assortment of required steps, optional steps, and information. To make things easier, we have highlighted the ''required steps'' with a red bar, like you see on the left of this message. Do not skip anything marked with a red bar, or else you will not have a perfectly configured instance.</span>

}}

=== Learn About Linux ===

Linux is an open-source operating system based on UNIX. Linux is highly versatile and is used in a wide range of applications.

If you are not a Linux guru, read the Linux guide to get started: [[Linux]]

=== Windows Users Only: Enable Windows Subsystem for Linux (WSL) or Install Cygwin ===

You will need WSL or Cygwin in order to log into and configure your EC2 instance. Cygwin is already installed on the CEC machines. If you're using Windows 10, you can use WSL, which is an officially-supported way of accessing a Linux terminal on Windows. If not, you'll have to use Cygwin. Instructions for installing either on your personal machine are located in [[Workflow#Cygwin|the Workflow guide]].

Mac OS X users should located the "Terminal" application, which is installed by default; you will need this in the coming steps.

=== Create an AWS EC2 Instance ===

Once you understand [[Linux]], you need to set up your Amazon EC2 Instance. Use the AWS article to guide you through the process: [[Amazon Web Services]]

=== Configure SSH ===

Follow the instructions in [[SSH#SSH Configuration|the SSH guide]] to do the following:

# Create your own SSH key pair
# Log in as the default user to your EC2 instance
# Create a new user
# Give the new user your SSH public key

From now on, when you need to log into your EC2 instance as an administrator, just pop up a terminal and run:

$ ssh <your-instance-username>@<ec2-xx-xx-xx-xx.compute-1.amazonaws.com>

To save from typing this command in every time you want to access your server, you can set an '''alias''' in your terminal.

In the '''Mac''' terminal, vim ~/.zshrc and input the following command:
$ alias aws='ssh <your-instance-username>@<ec2-xx-xx-xx-xx.compute-1.amazonaws.com>'
Now, whenever you type aws in your terminal, it will type your ssh command.

In '''Cygwin''', vim ~/.bash_profile and input the following:
export aws="ssh <your-instance-username>@<ec2-xx-xx-xx-xx.compute-1.amazonaws.com>"
And you run this command by typing '''$aws'''
Also note that you may have to restart your terminal for this to take effect, or put the command in your ~/.bashrc file

=== Install Essential Packages ===

Recall that software installation in Linux instances is usually handled through a package manager, like '''yum'''. For more information, refer to [[Linux#Repository-Based Package Managers|the Linux guide]].

To make sure everything is working properly, '''install your first package(s) according to the ''Essential Packages'' subsection in the Linux guide.'''

=== Set the Timezone ===

'''Set your server to use US Central time (America/Chicago).''' Instructions are in the Linux guide: [[Linux#Synchronizing Date and Time]]

=== Set Up the Apache Web Server ===

You need to '''install the Apache web server on your EC2 instance'''. Refer to [[Apache|the Apache guide]].

In order for your web server to be accessible, you need to '''open up Port 80 on your EC2 instance'''. Instructions are in the AWS guide: [[Amazon Web Services#Enabling Web Access to your EC2 Instance]]

'''Enable the UserDir module in Apache.''' Instructions are in the Apache guide: [[Apache#The UserDir Module]]

=== Install PHP ===

You need to '''install PHP on your EC2 server'''. Instructions are at the top of the PHP guide: [[PHP]]

You need to '''configure PHP to show errors'''. Instructions are immediately below installation in the PHP guide.

=== Learn about the PHP Language ===

Before you can continue, you will need to make yourself acquainted with PHP language components. Read the PHP guide for an overview: [[PHP#PHP Language Components]]

=== Set Up your Workspace ===

You are now ready to set up the workspace that you will be using for the rest of the semester. Follow the instructions in [[Workflow]] to:

# Make an account on GitHub
# Setup a repo using GitHub Classroom
# Clone your CSE 330 repo from GitHub
# Install a text editor such as '''VSCode''' to write your assignments with
# Either install an FTP client, or use SFTP through command line, to transfer files from your local machine to your instance

=== Calculator ===

'''Make a calculator using PHP and an HTML form.''' The form should have two inputs. The form should submit a GET request either back to the same page or to a different results page. The calculator should support addition, subtraction, multiplication, and division by means of a radio button group on the page. (Consider edge cases!)

=== Git Tutorial ===

Complete all of the levels of the Main and Remote sections of [https://learngitbranching.js.org/ this Git tutorial]. After you finish the tutorial, take two screenshots showing the completion of all of the levels of the Main and Remote sections. Place these two screenshots in your Module 2 Individual repo.

== Group Project ==

You will work in pairs (that means you and one other person) on this project.

<span style="font-size:1.3em; line-height:1.2em; color:blue;">In this module, and all future modules, the group portion builds off of material you learn in the individual portion. You should therefore complete the individual portions prior to working on the group portions. (Trust us: it will make your life easier!)</span>

'''Important Reminder:''' frequently commit your work to your repository as a backup!

=== Simple File Sharing Site ===

You will be making a simple file sharing site that supports uploading, viewing, and deleting files associated with various users. Details:

* You should have a file named '''users.txt''' stored in a secure place on your filesystem. It should have at least three usernames, with one username per line.
*: Passwords add an extra layer of complexity (encryption) that we will cover in Module 3. You will not earn credit for implementing passwords as your creative portion for this module.
* Users of the file sharing site should be able to enter their username and then log in.
*: You may implement logins using session variables, or you may implement them using a GET parameter that is passed between pages. You will need to always use sessions starting in Module 3.
* Users should see a list of all files associated with their username after the log in.
* Users should be able to view, upload, and delete files associated with their username.
* The URI should NOT reveal the internal file structure of your web site.
** Example of what not to do: http://www.example.com/download.php?file=/home/paul/file.txt
** Another bad example: http://www.example.com/~user/file.txt
** Good example: http://www.example.com/view.php?user=music&name=file.txt
*: In general, don't reveal any internal information about the site.
*: '''Food for thought:''' Given that Apache processes (with the exception of the main process) do not run as root, but rather as the user apache, how can you allow both your user account and the php scripts to read and write the
*:necessary files?
*:: ''Hint:'' You shouldn't be storing the user-uploaded files in public_html (that is only the place for your own web-page files),
*::''Hint number 2:'' you will need to use chown to change permissions to let user apache have permissions on the user-files (that '''are not''' in the public_html folder). If you don't know what this does or how to do this, look at the [[Apache]] page
* You should customize your file sharing site by implementing an additional feature or two as part of the '''creative portion'''.

You will probably find the PHP guide on this wiki to be helpful: [[PHP#Other PHP Tips]]

=== Web Security and Validation ===

Your project needs to demonstrate that thought was put into web security and best practice. For more information, see this week's Web Application Security guide: [[Web Application Security, Part 1]]

In particular:

* '''Your application needs to follow the conventions of FIEO (Filter Input and Escape Output)'''. Things you should filter include file names, usernames, and so on.
* '''Your project must pass the W3C Validator with no errors or warnings'''. For more information, see the HTML and CSS guide: [[HTML and CSS#Validation]]

== Grading ==

We will be grading the following aspects of your work. There are 120 points total.

'''Assignments (including code) must be committed to GitHub by the end of class on the due date. Commit early and often. Failing to commit by the end of class on the due date will result in a 0.

_____________

# '''Cloud Instance and Web Server (30 Points):'''
#* You can SSH into your instance using your own custom username and SSH key. Put a screenshot in your repository of your terminal SSH-ing into your instance (15 points)
#* Your custom username is able to perform SUDO commands. Put a screenshot in your repository of you successfully executing a command with sudo (5 points)
#* The time zone is correct on your instance. Put a screenshot in your repository of the results of the `date` command (5 points)
#* The UserDir Apache module is working. Place a file in /home/<username>/public_html and link to the file in your README.md (5 points)
# '''PHP Calculator (15 Points):'''
#* The calculator is able to perform all four functions with floating point numbers (2 points each, 8 points in total)
#* Each function is given as a radio button choice (4 points)
#* The calculator page passes the W3C HTML validator (3 points)
#* '''''Make sure you have a README.md file in your individual repo with the link to your calculator on your instance.''''' See the Creative Portion section for more information about .md files.
# '''Git Tutorial (20 Points):'''
#* Two screenshots showing the completion of all of the levels of the Main and Remote sections placed in your individual assignment repository.
# '''File Sharing Site (40 Points):'''
#* '''''File Management (25 Points):'''''
#** Users should not be able to see any files until they enter a username and log in (4 points)
#**: ''Remember that users.txt should be stored in a secure location on your filesystem. That is, '''you should not be able to type any URL into your browser and see the raw users.txt file!'''''
#** Users can see a list of all files they have uploaded (4 points)
#** Users can open files they have previously uploaded (5 points)
#**: ''Note: Users should be able to open not only plain text files but also other file formats: images, spreadsheets, etc.''
#** Users can upload files (4 points)
#**: ''Note: Like users.txt, uploaded files should be stored in a secure location on your filesystem. That is, '''do not keep your uploads directory underneath a directory served by Apache!'''''
#** Users can delete files. If a file is "deleted", it should actually be removed from the filesystem (4 points)
#** The directory structure is hidden. Users should not be able to access or view files by manipulating a URL. (2 points)
#** Users can log out (2 points)
#**: ''Note: If using session variables, you must ''actually'' log out the user by destroying their session; i.e., don't just redirect them to the login screen.''
#* '''''Best Practices (10 Points):'''''
#** Code is well formatted and easy to read, with proper commenting (4 points)
#** The site follows the FIEO philosophy (3 points)
#** All pages pass the W3C validator (3 points)
#* '''''Usability (5 Points):'''''
#** Site is intuitive to use and navigate (4 points)
#** Site is visually appealing (1 point)
# '''Creative Portion (15 Points)''' (see below)
#* '''''Make sure you have a README.md file in your group repo with the following:'''''
#** An .md file is a plain text file called a markdown file [https://guides.github.com/features/mastering-markdown/ ] .
#***The contents of your README.md file are shown on your repo, making it easier for us to grade
#*** You may style the page with bolding, hyperlings, images, block quotes and lists
#*** [http://dillinger.io/ This online markdown editor] makes it easy to learn this format
#** The link to your file sharing site. '''''We need this to grade your work.'''''
#** A brief description of what you did for your creative portion
#** Any additional login details needed for the TA

=== Creative Portion ===

This module and all future modules will require that you invest some time into creating additional features for your group project. Plan to invest at the very least 60 minutes of your time into the creative portion. In the above rubric, viewing files is worth five and uploading is worth four. The creative portion is worth fifteen points; it should be a very sizable part of your project.

* The creative portion is an opportunity for you to learn the material of your own interest.
* You will not earn credit for a creative-portion feature that simply rehashes something you've already done in a previous module.

If you need ideas for a creative portion, or if you want to know whether or not your creative portion idea is "hard enough", ask a TA.

[[Category:Module 2]]
[[Category:Modules]]

VS Code

2023-01-10T06:33:59Z

Zach: /* SFTP Integration (Shipping files to your server) */

== Getting Started With Each Module ==

* Create Github repo using the CSE 330 repo creator (Check Piazza for the link)
* Navigate to the created repo in Github
* Clone the repo via https to a dedicated folder on your computer

[[File:File-Structure.png|400px|center]]

* In VS Code, File > Open the Spring2019-module1-individual-STUDENTID '''folder''' that you cloned to your computer

If you followed the above steps correctly, the folder's Git should be configured to your Github repository! I highly recommend creating your Github repo, cloning the folder to a designated spot, and keeping all modules separated for the semester as shown in the image above.

== Module 1 -- Using the Git Integration ==
'''Note: If you're using Windows, you may need to install Git on your machine first. https://git-scm.com/download/win . You can check if you have Git running by typing git --version in your terminal'''

* First, open the File Explorer tab on the left, and right click to create a new file, birthday.html
[[File:New-file.png|400px|center]]
* Ensure VS Code has it marked as an html file (see the file type next to the smiley face)
* Type '''doc''' followed by enter, and allow VS Code to generate a basic HTML file for you (use VS Code to your advantage! It can do a lot of your coding for you).
* Save the file, you should see a popup next to the Version Control symbol on the left. VS Code has detected that your project is different than what it is in your remote Github repo.
* To commit, simply type a message in the box and click the check mark above it. To push to your Github repo, click the 3 dots next to the check mark and select 'push'
* Note this is also where you can pull from your remote.

==== Bonus - Module 1 ====
* If you like your text editor to look pretty, you can go to the Extensions tab and search for different themes. (Check out the Material theme!)
* You can view your settings by clicking CMD + , (Mac) or Ctrl + , (Windows). Search for Theme and choose one that works for you!

== Module 2 ==
=== DISCLAIMER ===
{{RequiredInstructions|content=

<span style="color:red">There are two SFTP VSCode extensions of the same name. One of them is depreciated. Use the extension distributed by Natizyskunk.</span>

}}

=== SFTP Integration (Shipping files to your server) ===
# Select a folder to use for your work and open it in VSCode. When editing files on your instance, the file will be saved locally in this location for editing and reuploaded
# Open the Extensions tab and search for 'SFTP'. Download the version distributed by "Natizyskunk".
# To use commands in VS Code, type '''Cmd + Shift + P''' (Mac), or '''Ctrl + Shift + P''' on windows.
# Type SFTP and open the sftp config command.

Your config.json file should look like the following:
{
"name": "<My Instance>",
"host": "<ec2-xxxxx-xxxxxx-xxxxx.amazonaws.com>",
"protocol": "sftp",
"port": 22,
"username": "<custom instance username>"
"remotePath: "/home/<custom instance username>/",
"uploadOnSave": true,
"privateKeyPath": "</path/to/.ssh/id_rsa>"
}
- Note the uploadOnSave field is optional, but can be nice!
- Check the VS Code output for errors in your config.

Additionally: You should notice an SFTP tab on the side of VS Code. Here, you can view the file structure on your server.]

'''Preventing "Edit in Local" when editing files'''

In order to edit locally, you'll have to right click on the file you wish to edit and select "Edit in local" every time. To avoid this, you can:
# Open vscode settings by File >> Preferences >> Settings
# In settings, search for "downloadWhenOpenInRemoteExplorer" and toggle it on
# restart VScode.

=== Testing Everything Locally ===
If you would like to avoid having to frequently upload your changes to your instance in order to test your changes, you can optionally set up a local PHP server.

'''To run PHP locally'''

'''Windows'''
# Download WAMP http://www.wampserver.com/en/download-wampserver-64bits/
# Place your project repositories in Wamp's www/ directory, and navigate to http://localhost in your browser. You should see your projects listed here, run php files, and use phpmyadmin.

'''Mac'''
# Download Homebrew. Homebrew is a great package manager for OSx and we will be using it in future modules.
# Type the following in your terminal and follow instructions
/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
# In your terminal, type '''php -v'''. If your version is less than 7, update with brew. '''brew install php'''
# In VS Code, search for the extension '''Php Server'''. After installing, use VS Code's command pallette with CMD+Shift+P , search for PHP, and select start server. You should be able to view your project in your localhost

VS Code

2023-01-10T06:33:36Z

Zach: /* SFTP Integration (Shipping files to your server) */

== Getting Started With Each Module ==

* Create Github repo using the CSE 330 repo creator (Check Piazza for the link)
* Navigate to the created repo in Github
* Clone the repo via https to a dedicated folder on your computer

[[File:File-Structure.png|400px|center]]

* In VS Code, File > Open the Spring2019-module1-individual-STUDENTID '''folder''' that you cloned to your computer

If you followed the above steps correctly, the folder's Git should be configured to your Github repository! I highly recommend creating your Github repo, cloning the folder to a designated spot, and keeping all modules separated for the semester as shown in the image above.

== Module 1 -- Using the Git Integration ==
'''Note: If you're using Windows, you may need to install Git on your machine first. https://git-scm.com/download/win . You can check if you have Git running by typing git --version in your terminal'''

* First, open the File Explorer tab on the left, and right click to create a new file, birthday.html
[[File:New-file.png|400px|center]]
* Ensure VS Code has it marked as an html file (see the file type next to the smiley face)
* Type '''doc''' followed by enter, and allow VS Code to generate a basic HTML file for you (use VS Code to your advantage! It can do a lot of your coding for you).
* Save the file, you should see a popup next to the Version Control symbol on the left. VS Code has detected that your project is different than what it is in your remote Github repo.
* To commit, simply type a message in the box and click the check mark above it. To push to your Github repo, click the 3 dots next to the check mark and select 'push'
* Note this is also where you can pull from your remote.

==== Bonus - Module 1 ====
* If you like your text editor to look pretty, you can go to the Extensions tab and search for different themes. (Check out the Material theme!)
* You can view your settings by clicking CMD + , (Mac) or Ctrl + , (Windows). Search for Theme and choose one that works for you!

== Module 2 ==
=== DISCLAIMER ===
{{RequiredInstructions|content=

<span style="color:red">There are two SFTP VSCode extensions of the same name. One of them is depreciated. Use the extension distributed by Natizyskunk.</span>

}}

=== SFTP Integration (Shipping files to your server) ===
# Select a folder to use for your work and open it in VSCode. When editing files on your instance, the file will be saved locally in this location for editing and reuploaded
# Open the Extensions tab and search for 'SFTP'. Download the version distributed by "Natizyskunk".
# To use commands in VS Code, type '''Cmd + Shift + P''' (Mac), or '''Ctrl + Shift + P''' on windows.
# Type SFTP and open the sftp config command.

Your config.json file should look like the following:
{
"name": "<My Instance>",
"host": "<ec2-xxxxx-xxxxxx-xxxxx.amazonaws.com>",
"protocol": "sftp",
"port": 22,
"username": "<custom instance username>"
"remotePath: "/home/<custom instance username>/",
"uploadOnSave": true,
"privateKeyPath": "</path/to/.ssh/id_rsa>"
}
- Note the uploadOnSave field is optional, but can be nice!
- Check the VS Code output for errors in your config.

Additionally: You should notice an SFTP tab on the side of VS Code. Here, you can view the file structure on your server.]

'''Preventing "Edit in Local" when editing files'''
In order to edit locally, you'll have to right click on the file you wish to edit and select "Edit in local" every time. To avoid this, you can:
# Open vscode settings by File >> Preferences >> Settings
# In settings, search for "downloadWhenOpenInRemoteExplorer" and toggle it on
# restart VScode.

=== Testing Everything Locally ===
If you would like to avoid having to frequently upload your changes to your instance in order to test your changes, you can optionally set up a local PHP server.

'''To run PHP locally'''

'''Windows'''
# Download WAMP http://www.wampserver.com/en/download-wampserver-64bits/
# Place your project repositories in Wamp's www/ directory, and navigate to http://localhost in your browser. You should see your projects listed here, run php files, and use phpmyadmin.

'''Mac'''
# Download Homebrew. Homebrew is a great package manager for OSx and we will be using it in future modules.
# Type the following in your terminal and follow instructions
/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
# In your terminal, type '''php -v'''. If your version is less than 7, update with brew. '''brew install php'''
# In VS Code, search for the extension '''Php Server'''. After installing, use VS Code's command pallette with CMD+Shift+P , search for PHP, and select start server. You should be able to view your project in your localhost

VS Code

2023-01-10T06:32:31Z

Zach: /* SFTP Integration (Shipping files to your server) */

== Getting Started With Each Module ==

* Create Github repo using the CSE 330 repo creator (Check Piazza for the link)
* Navigate to the created repo in Github
* Clone the repo via https to a dedicated folder on your computer

[[File:File-Structure.png|400px|center]]

* In VS Code, File > Open the Spring2019-module1-individual-STUDENTID '''folder''' that you cloned to your computer

If you followed the above steps correctly, the folder's Git should be configured to your Github repository! I highly recommend creating your Github repo, cloning the folder to a designated spot, and keeping all modules separated for the semester as shown in the image above.

== Module 1 -- Using the Git Integration ==
'''Note: If you're using Windows, you may need to install Git on your machine first. https://git-scm.com/download/win . You can check if you have Git running by typing git --version in your terminal'''

* First, open the File Explorer tab on the left, and right click to create a new file, birthday.html
[[File:New-file.png|400px|center]]
* Ensure VS Code has it marked as an html file (see the file type next to the smiley face)
* Type '''doc''' followed by enter, and allow VS Code to generate a basic HTML file for you (use VS Code to your advantage! It can do a lot of your coding for you).
* Save the file, you should see a popup next to the Version Control symbol on the left. VS Code has detected that your project is different than what it is in your remote Github repo.
* To commit, simply type a message in the box and click the check mark above it. To push to your Github repo, click the 3 dots next to the check mark and select 'push'
* Note this is also where you can pull from your remote.

==== Bonus - Module 1 ====
* If you like your text editor to look pretty, you can go to the Extensions tab and search for different themes. (Check out the Material theme!)
* You can view your settings by clicking CMD + , (Mac) or Ctrl + , (Windows). Search for Theme and choose one that works for you!

== Module 2 ==
=== DISCLAIMER ===
{{RequiredInstructions|content=

<span style="color:red">There are two SFTP VSCode extensions of the same name. One of them is depreciated. Use the extension distributed by Natizyskunk.</span>

}}

=== SFTP Integration (Shipping files to your server) ===
# Select a folder to use for your work and open it in VSCode. When editing files on your instance, the file will be saved locally in this location for editing and reuploaded
# Open the Extensions tab and search for 'SFTP'. Download the version distributed by "Natizyskunk".
# To use commands in VS Code, type '''Cmd + Shift + P''' (Mac), or '''Ctrl + Shift + P''' on windows.
# Type SFTP and open the sftp config command.

Your config.json file should look like the following:
{
"name": "<My Instance>",
"host": "<ec2-xxxxx-xxxxxx-xxxxx.amazonaws.com>",
"protocol": "sftp",
"port": 22,
"username": "<custom instance username>"
"remotePath: "/home/<custom instance username>/",
"uploadOnSave": true,
"privateKeyPath": "</path/to/.ssh/id_rsa>"
}
- Note the uploadOnSave field is optional, but can be nice!
- Check the VS Code output for errors in your config.

Additionally: You should notice an SFTP tab on the side of VS Code. Here, you can view the file structure on your server.]

In order to edit locally, you'll have to right click on the file you wish to edit and select "Edit in local" every time. To avoid this, you can:
# Open vscode settings by File >> Preferences >> Settings
# In settings, search for "downloadWhenOpenInRemoteExplorer" and toggle it on
# restart VScode.

=== Testing Everything Locally ===
If you would like to avoid having to frequently upload your changes to your instance in order to test your changes, you can optionally set up a local PHP server.

'''To run PHP locally'''

'''Windows'''
# Download WAMP http://www.wampserver.com/en/download-wampserver-64bits/
# Place your project repositories in Wamp's www/ directory, and navigate to http://localhost in your browser. You should see your projects listed here, run php files, and use phpmyadmin.

'''Mac'''
# Download Homebrew. Homebrew is a great package manager for OSx and we will be using it in future modules.
# Type the following in your terminal and follow instructions
/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
# In your terminal, type '''php -v'''. If your version is less than 7, update with brew. '''brew install php'''
# In VS Code, search for the extension '''Php Server'''. After installing, use VS Code's command pallette with CMD+Shift+P , search for PHP, and select start server. You should be able to view your project in your localhost

VS Code

2023-01-10T06:20:17Z

Zach: /* DISCLAIMER */

== Getting Started With Each Module ==

* Create Github repo using the CSE 330 repo creator (Check Piazza for the link)
* Navigate to the created repo in Github
* Clone the repo via https to a dedicated folder on your computer

[[File:File-Structure.png|400px|center]]

* In VS Code, File > Open the Spring2019-module1-individual-STUDENTID '''folder''' that you cloned to your computer

If you followed the above steps correctly, the folder's Git should be configured to your Github repository! I highly recommend creating your Github repo, cloning the folder to a designated spot, and keeping all modules separated for the semester as shown in the image above.

== Module 1 -- Using the Git Integration ==
'''Note: If you're using Windows, you may need to install Git on your machine first. https://git-scm.com/download/win . You can check if you have Git running by typing git --version in your terminal'''

* First, open the File Explorer tab on the left, and right click to create a new file, birthday.html
[[File:New-file.png|400px|center]]
* Ensure VS Code has it marked as an html file (see the file type next to the smiley face)
* Type '''doc''' followed by enter, and allow VS Code to generate a basic HTML file for you (use VS Code to your advantage! It can do a lot of your coding for you).
* Save the file, you should see a popup next to the Version Control symbol on the left. VS Code has detected that your project is different than what it is in your remote Github repo.
* To commit, simply type a message in the box and click the check mark above it. To push to your Github repo, click the 3 dots next to the check mark and select 'push'
* Note this is also where you can pull from your remote.

==== Bonus - Module 1 ====
* If you like your text editor to look pretty, you can go to the Extensions tab and search for different themes. (Check out the Material theme!)
* You can view your settings by clicking CMD + , (Mac) or Ctrl + , (Windows). Search for Theme and choose one that works for you!

== Module 2 ==
=== DISCLAIMER ===
{{RequiredInstructions|content=

<span style="color:red">There are two SFTP VSCode extensions of the same name. One of them is depreciated. Use the extension distributed by Natizyskunk.</span>

}}

=== SFTP Integration (Shipping files to your server) ===
# Open the Extensions tab and search for 'sftp'. Download the first one.
# To use commands in VS Code, type '''Cmd + Shift + P''' (Mac), or '''Ctrl + Shift + P''' on windows.
# Type SFTP and open the sftp config command.

Your config.json file should look like the following:
{
"protocol": "sftp",
"host": "YOUR.EC2.DNS",
"port": 22,
"username": "YOUR USERNAME",
"privateKeyPath": "/path/to/your/id_rsa",
"remotePath": "/home/YOUR USERNAME/public_html/module2",
"uploadOnSave": true
}
- Note the uploadOnSave field is optional, but can be nice!
- Check the VS Code output for errors in your config.

Additionally: You should notice an SFTP tab on the side of VS Code. Here, you can view the file structure on your server.

=== Testing Everything Locally ===
If you would like to avoid having to frequently upload your changes to your instance in order to test your changes, you can optionally set up a local PHP server.

'''To run PHP locally'''

'''Windows'''
# Download WAMP http://www.wampserver.com/en/download-wampserver-64bits/
# Place your project repositories in Wamp's www/ directory, and navigate to http://localhost in your browser. You should see your projects listed here, run php files, and use phpmyadmin.

'''Mac'''
# Download Homebrew. Homebrew is a great package manager for OSx and we will be using it in future modules.
# Type the following in your terminal and follow instructions
/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
# In your terminal, type '''php -v'''. If your version is less than 7, update with brew. '''brew install php'''
# In VS Code, search for the extension '''Php Server'''. After installing, use VS Code's command pallette with CMD+Shift+P , search for PHP, and select start server. You should be able to view your project in your localhost

Python

2023-01-10T06:10:26Z

Zach: /* Python Script Files */

Languages like Java and C++ have lots of rules regarding variable types, syntax, return values, and so on. These can be cumbersome when you are trying to write short, quick scripts to perform tasks. This is where a ''scripting language'' comes into play.

'''Python''' is a language well-suited to rapid prototype development. It is generally an ''interpreted language'', which means that, generally speaking, a Python program is converted line by line into instructions for the computer to execute as the program runs, instead of having a separate compilation step beforehand {{ref|interpreted}}. The syntax is clean, and it is usually clear at first glance what is going on when you write in Python.

= Installation =

Python may already be installed on your system. To see whether or not it is, enter the command

<source lang="bash">
$ python3 --version
</source>
If it tells you a version of Python '''greater than or equal to 3.6''' then you're good to go. If not, you need to do a quick package install to get it up and running. Using yum, it's just <code>sudo yum install python36</code>.

If you'd like to be able to just type <code>python</code> instead of <code>python3</code>, you can add the following line to the <code>~/.bashrc</code> file on your instance:

<source lang="bash">
alias python="/usr/bin/python3"
</source>

==== Python 2 vs Python 3 ====

In 2008, a new version of Python – Python 3 – was released, with a number of changes that made it incompatible with the previous version, Python 2. For several years, the Python community was slow to make the change to Python 3, but today, Python 3 is ''the'' standard for new code. The ''only'' reason to write Python 2 code today is because you have to work in an environment or with libraries that do not support Python 3, and that situation is becoming increasingly rare.

'''For CSE 330 (and this guide), you ''must'' use Python 3.6 or greater.'''

=== Pip ===

Linux distributions have package managers like ''Apt'', ''Yum'', and ''YaST''. PHP has a package manager named ''PEAR''. It's now time to introduce Python's leading package manager: '''pip'''.

Once you have pip installed, you can use it to install Python packages. Use the <code>pip-3.6</code> command:

<source lang="bash">
$ pip-3.6 install <package_name> # RHEL
</source>

If you'd like to be able to just type <code>pip</code> instead of <code>pip-3.6</code>, you can add the following line to the <code>~/.bashrc</code> file on your instance:

<source lang="bash">
alias pip="/usr/bin/pip-3.6"
</source>

= Running Python =

There are two common ways to run Python code: via the console, and via a Python script file.

=== The Python Console ===

The Python console enables you to experiment with code without opening a text editor. To enter the Python console, simply type the ''python'' command at the terminal:

<source lang="bash">
$ python3
</source>
To leave the interactive console, either type <code>quit()</code> or press <code>Ctrl-D</code> (on both Mac and Windows).

=== Python Script Files ===

You can also save Python script files for later use. The extension for Python scripts is <code>*.py</code>. To run a script file, simply feed its path as an argument to the <code>python</code> command in Terminal:

<source lang="bash">
$ python3 <my_script>.py
</source>

= Python Syntax and Language Components =

This section contains a very brief overview of Python syntax. For a more comprehensive introduction, see [http://docs.python.org/tutorial/ the Python docs].

=== Python's Type System ===

Python is a dynamically-typed, strongly-typed, and duck-typed language. What does that mean? Well, let's take those categories one by one:

<ol>
<li>
'''Dynamic typing''', in contrast to ''static typing'' means that the types of objects can be changed. In Java, for instance, if you tried to write the following code:

<source lang="java">
int c = 2;
c = "fish";
</source>

Your code would fail to compile, because the Java compiler wouldn't allow you to change the type of the variable <code>c</code> from <code>int</code> to <code>String</code>.

Python, however, is just fine with that. You can write:

<source lang="python">
c = 2
c = "fish"
</source>

And Python will happily do what you ask.
</li>

<li>
'''Strong typing''', in contrast to ''weak typing'', means, roughly, that a language won't implicitly convert types for you. So far, you've used a lot of PHP, which is very weakly typed. For example:

<source lang="php">
echo "10" . 2;
</source>

In PHP, this works just fine. The PHP interpreter sees that <code>2</code> is an int, and silently converts it to a string for you, so that <code>"10". 2</code> equals <code>"102"</code>. The problem with weak typing is that it can make it unclear to the programmer why things are happening the way they are. For instance, in PHP, the expression <code>"10" == 10</code> is true, which can be less than obvious. Python, however, tries to disallow that sort of thing:

<source lang="python">
"10" + "10" # Allowed – concatenates the two strings.
10 + 10 # Allowed – adds the two ints.
"10" + 10 # Not allowed – raises a TypeError
</source>

Perhaps the one exception to this rule is the relationship between numeric types; it's perfectly fine to add a float to an int, for example.
</li>

<li>
'''Duck-typing''' is a little bit different than the other categories, because it doesn't really have a direct opposite. Essentially, duck-typing is the philosophy that "If it walks like a duck, and quacks like a duck, it's a duck." In programming terms, that means that we often don't care what type an object is, as long as it can do the things we ask.

Let's look at an example:

<source lang="python">
# This function doubles the value it's passed.
def double(thing):
return thing + thing
double(10) # Returns 20
double("fish") # Returns "fishfish";
</source>

If this were Java, we'd have to write two different methods here: one for things of type <code>int</code>, and one for things of type <code>String</code>. Since this is Python, though, we don't have to do that. Instead, Python lets us say, essentially: "This function uses the <code>+</code> operator, so you can pass whatever object you want to the function, as long as the function supports the <code>+</code> operator".

To quote [https://groups.google.com/forum/?hl=en#!msg/comp.lang.python/CCs2oJdyuzc/NYjla5HKMOIJ Alex Martelli], one of the key people in the Python project,

<blockquote>
"[Don't] check whether it IS-a duck: check whether it QUACKS-like-a duck, WALKS-like-a duck, etc, etc, depending on exactly what subset of duck-like behaviour you need."
</blockquote>

</li>
</ol>

=== Basic Types ===

In Python, the basic scalar types are:

# <code>str</code> – a string.
# <code>int</code> – an integer object with unlimited digits.
# <code>bool</code> – a Boolean value.
# <code>float</code> – a floating point number; generally equivalent to <code>double</code> in Java.
# <code>bytes</code> – represents a collection of bytes; you're unlikely to use this often.

Casting is done in the C style - to parse an <code>int</code> from a string, for instance, you'd write <code>int("123")</code>.

=== Basic Control Flow ===
In Python, blocks of code are delineated using colons and indentation instead of curly braces. '''''That means that proper indentation is mandatory.''''' Every time you use a colon, you must indent one step further, and every time a block ends, you must unindent a step.

Conditional statements are done using the <code>if</code>, <code>elif</code>, and <code>else</code> operators, like in the example below - pay attention to the indentation:

<source lang="python">
cookies_left = 10
cake_slices_left = 3
if cookies_left > 10:
print("Lots of cookies!")
elif cookies_left > 8:
print("Plenty left!")
elif cookies_left == 5:
print("Halfway done!")
else:
if cake_slices_left == 3:
print("Lots of cake!")
elif cake_slices_left == 2:
print("One slice gone!")
else:
print("Little to no cake left!")
</source>

=== An Example Python Script ===

In my mind, there's no better way to learn Python than to be immersed in a simple example script.

<source lang="python">
print("Hello World")

fruits = ["apple", "banana", "cherry", "date"]
for fruit in fruits:
print(f"I always love to eat a fresh {fruit}")

# Map the fruits list over to a new list containing the length of the fruit strings:
fruit_size = [len(fruit) for fruit in fruits]

avg_fruit_size = sum(fruit_size) / len(fruit_size)
print(f"The average fruit string length is {avg_fruit_size:.2f}.")
</source>

Some things to notice:

* Printing is achieved using the <code>print()</code> function.
* Comments start with a pound symbol <code>#</code>.
* We can transform/map a list to a new list in just one line!

For some more examples, see [https://wiki.python.org/moin/SimplePrograms the Python wiki].

=== A Few Gotchas ===

Here are a few things you may not expect when starting out with Python:

<ol>

<li>
Strings can be delimited using either single quotes (<code>'</code>) or double quotes (<code>"</code>), so <code>'Python'</code> is the same as <code>"Python"</code>. You can also use three sets of quotes (single or double again) to delineate a multiple-line string, like so:
<source lang="python">
long_string = """This
is my
very long
multiple-line
string"""
</source>
</li>

<li>
To compare the equality of two objects, use the <code>==</code> operator. To compare the identity of two objects (i.e. whether or not they are literally the same object, at the same place in memory), use the <code>is</code> operator. These correspond, respectively to <code>.equals()</code> and <code>==</code> in Java.
</li>

<li>
The <code>/</code> operator performs floating-point division, not integer division, even when both its arguments are integers. If you want to do integer division, use the <code>//</code> operator. Let's look at an example:

<source lang="python">
9 / 10 == 0.9 # Floating-point division
9.0 / 10.0 == 0.9 # Floating-point division
9 // 10 == 0 # Integer division
9.0 // 10.0 == 0.0 # Integer division, even though the resulting value is a float
</source>
</li>

<li>
As discussed in the section [[#Python's_Type_System|Python's Type System]], you can't combine strings and numbers using the <code>+</code> operator. Instead, you'll have to use one of the following methods of string formatting:

<source lang="python">

# Technique 1 – printf-style Formatting
# This is a classic style of string formatting, used in many programming languages, including in Java's String.format.
"You scored %d out of %d!" % (score, questions)

# Technique 2 – str.format
"You scored {} out of {}!".format(score, questions)

# Technique 3 – f-strings
# These were only introduced in Python 3.6 but they're very convenient.
f"You scored {score} out of {questions}!"

# Technique 4 - String Concatenation
# This is fine for small things, but for more complicated stuff, use one of the other techniques.
"You scored " + str(score) + " out of " + str(questions) + "!"
</source>
</li>
</ol>

=== Functions ===

Define functions using the <code>def</code> keyword:

<source lang="python">
def hello(name):
print(f"Hello, {name}!")

</source>

Invoke functions using parentheses:
<source lang="python">
hello("Batman")
hello("Superman")
</source>

Functions can have optional arguments; however, these must come after any required arguments:
<source lang="python">
def hello(name="User"):
print(f"Hello, {name}!")

hello() # Prints "Hello, User!"
hello("Sarah") # Prints "Hello, Sarah!"
</source>

Functions are just regular objects in Python, meaning you can do stuff like this:

<source lang="python">
def hello(name="User"):
print(f"Hello, {name}!")

greeting = hello

greeting("Siwei") # Prints "Hello, Siwei!"
</source>

=== Collection Types ===
Python has a few basic, built-in collection types. They each have their own special use cases, but they share a lot of similarities. All built-in data structures in Python are zero-indexed, for instance.

To obtain the number of elements in a collection, use the <code>len()</code> function, like this:

<source lang="python">
pets = ['Dog', 'Cat', 'Fish'] # This is a list; you'll learn about them below.

len(pets) # Returns 3
</source>

To check if a collection contains an item, use the <code>in</code> operator:
<source lang="python">
pets = ['Dog', 'Cat', 'Fish'] # This is a list; you'll learn about them below.

'Dog' in pets # Returns True
</source>

Helpfully, the Python Software Foundation maintains a page containing the asymptotic complexities of important operations on major Python data structures. It's available [https://wiki.python.org/moin/TimeComplexity here].

==== Lists ====

Unlike languages like Java and C, Python's array-like type doesn't have fixed length. Instead, a <code>list</code> is a dynamic array (like Java's <code>ArrayList</code>) of objects of any type. Here's an example:

<source lang="python">
fruits = ["Apple", "Pear", "Orange"] # Lists can be creating by placing their items between square brackets

random_items = ['Apple', 12312, 2.0, [1, 2, 3]] # Lists can contain objects of different types - even other lists!

pets.append('Turtle') # Adds 'Turtle' to the end of pets
</source>

For the next few examples, we'll be using the following list:

<source lang="python">
pets = ['Dog', 'Cat', 'Fish', 'Turtle', 'Lizard', 'Snake']
</source>

To access or assign elements of a list, use the <code>[]</code> operator.

<source lang="python">
pets[0] # Returns 'Dog'
pets[1] = 'Cow' # Sets pets[1] to 'Cow'
pets[1] = 'Cat' # Sets pets[1] back to 'Cat'
</source>

You can also access or assign lists using negative indices, which will return elements starting from the end of the list:

<source lang="python">
pets[-1] # Returns 'Snake'
pets[-2] # Returns 'Lizard'
</source>

You can also use 'slice notation' to obtain or assign subsets of lists:
<source lang="python">
pets[start:end] # Returns items from index start to index end - 1.
pets[start:] # Returns items from index start to the end of the list, inclusive.
pets[:end] # Returns items from the beginning of the list to index end - 1.
pets[:] # Returns every item in the list
</source>

There's also a third, optional argument: <code>step</code>, which lets you jump over values:
<source lang="python">
pets[start:end:step] # Returns items from index start to index end - 1, jumping step values at a time

# Examples:
pets[1:5:2] # Returns items from index 1 to index 4, jumping 2 at a time, so ['Cat', 'Turtle']
pets[::3] # Returns all items in the list, jumping 3 at a time, so ['Dog', 'Turtle']
pets[::-1] # Returns all items in the list, jumping -1 at a time. This reverses the list, so ['Snake', 'Lizard', 'Turtle', 'Fish', 'Cat', 'Dog']
</source>

Lists have all the performance characteristics you'd expect of a dynamic array - constant-time indexing, amortized constant-time appends, and so on.

==== Tuples ====

Python has a special data type called <code>tuple</code>, which is exactly the same as a list except for the fact that it cannot be modified, so you can't append or replace items.

<source lang="python">
cities = ('St. Louis', 'Los Angeles', 'Seattle') # Tuples are defined using parentheses.
numbers = 1, 2, 3 # You can also omit the parentheses if it's unambiguous
single_item_tuple = (1,) # To make a tuple with only one item, put a comma after it
</source>

Tuples also enable you to have multiple return values from a function:

<source lang="python">
def compute_length(string):
str_len = len(string)
if str_len < 5:
return str_len, "short"
elif str_len < 40:
return str_len, "medium"
return str_len, "long"

length, description = compute_length("Four score and seven years ago")
print(f"The {description} string is {length} characters long.")
</source>
The above example also demonstrates Python's if...elif...else conditional structure.

==== Dictionaries ====

Python has another datatype called a '''dictionary''' (or <code>dict</code>, for short), which are like ''Maps'' in Java, ''associative arrays'' in PHP, and ''object literals'' in JavaScript (coming up soon in Module 6). Essentially, they enable you to use any immutable object as the key in your data structure. <code>dict</code> objects can be created using either the <code>dict</code> function, or by placing items between curly braces, separating keys and values using colons:

<source lang="python">
fruits_in_bowl = {
'apple': 4,
'banana': 2,
'cherry': 0,
'date': 12
}

apple_count = fruits_in_bowl['apple'] # Equals 4

fruits_in_bowl['pear'] = 9 # Sets 'pear' in the dictionary to 9.

for fruit, num in fruits_in_bowl.items():
print(f"There are {num} {fruit}(s) in the bowl." % (num, fruit))
</source>
If you only want the keys from a dictionary, you can just iterate over it directly, like this:

<source lang="python">
for fruit in fruits_in_bowl:
print(fruit)
</source>
If you want only the values but not the keys from a dictionary, use <code>my_dictionary.values()</code>.

==== Sets ====
A <code>set</code> is an unordered collection of unique items, with constant-time membership checking.
<source lang="python">
fruits = {'Apple', 'Apple', 'Apple', 'Pear'} # The set removes duplicates, so it will only contain {'Apple', 'Pear'}
fruits = set(['Apple', 'Apple', 'Apple', 'Pear']) # This line transforms a list into a set
print('Apple' in fruits) # Constant-time!
</source>

=== Loops ===

In Python, while loops are written like this:

<source lang="python">
i = 0
while i < 10:
i += 1
</source>
For loops are written like this:

<source lang="python">
pets = ['Dog', 'Cat', 'Fish']

for pet in pets:
print(pet)
</source>
To iterate over a range of numbers, you can use the convenient <code>range</code> function, like so:

<source lang="python">
for i in range(20, 30): # If you omit the first argument to range, it's assumed to be zero.
print(i)
</source>
Note that <code>range</code> provides a closed-open range. That is to say, it is inclusive of its lower bound and exclusive of its upper bound. To be very explicit about it:

<source lang="python">
list(range(3, 6)) == [3, 4, 5]
</source>
Unlike in other languages you may know, the following is not good practice in Python:

<source lang="python">
pets = ['Dog', 'Cat', 'Fish']

# BAD PRACTICE. DO NOT USE.
for i in range(len(pets)):
pet = pets[i]
</source>
Instead, if you need to access both a variable and its index, use the <code>enumerate</code> function, which allows you to iterate through a group of items, along with their indexes, like so:

<source lang="python">
pets = ['Dog', 'Cat', 'Fish']

for index, pet in enumerate(pets):
print(f"{pet} at index {index}")
</source>

=== Generator Expressions and Comprehensions ===

A generator is a special kind of object that, basically, represents a collection of items using a value and a function for generating the next value in the collection. In other words, it's a way of creating something that behaves like a list, but which only generates one value at a time, and discards a value when it's done with it. Say, for example, you wanted to do an operation on 10,000,000,000,000 numbers. You could put them all in a list, then iterate over the list, but that would take a lot of memory, plus you'd have to create and fill the entire list before beginning to actually work on your problem. Instead, you'd probably be better off using a generator; telling the computer, essentially: “The starting value is 0. When I ask for the next number in the collection, add 1 to current value and give me that.”

A generator expression is an expression that is evaluated to create a generator. They're super useful. Generator expressions can take two forms – they can be used with and without a condition:

<source lang="python">
(EXPRESSION for VARIABLE in ITERABLE) # Unconditional
(EXPRESSION for VARIABLE in ITERABLE if CONDITION) # Conditional
</source>
For our purposes, you can think of an iterable as anything you can do a for loop over. Let's take a look at a generator expression in use:

<source lang="python">
double_even_digits = (number * 2 for number in range(10) if number % 2 == 0)
</source>
That line says that <code>double_even_digits</code> is every even number between 0 and 10 multiplied by two. Simple enough, right? If we wanted to use all of the numbers, not just the even ones, we could simply remove the if statement at the end of the expression.

Here's the thing, though: if you were to print <code>double_even_digits</code>, instead of a nice bunch of numbers, you'd get something that looks like this:

<code><generator object <genexpr><code>at 0x107fc92b0></code>

That's because generators are lazy! They don't generate their values until those values are needed, and they discard values once they're no longer needed. As a result, '''you can only iterate over a generator once'''. In this case, you can't print out the whole generator, because it hasn't generated all the values yet. You can, however, iterate over the generator, and print its values one at a time, like so:

<source lang="python">
for number in double_even_digits:
print(number)
</source>
You can also make a list out of the generator, which will go through all the items in the generator and put them in a list, like so:

<source lang="python">
even_number_list = list(double_even_digits)
</source>
That's actually a great segue to the second topic of this section: comprehensions. A comprehension is just a prettier way to make lists, sets, and dicts out of a generator expressions.

<source lang="python">
# If you want a list:
list(i for i in range(10) if i % 2 == 0) # Don't do this
[i for i in range(10) if i % 2 == 0] # Use a list comprehension instead!

# If you want a set:
set(i for i in range(10) if i % 2 == 0) # Don't do this
{i for i in range(10) if i % 2 == 0} # Use a set comprehension instead!

# If you want a dict:
dict((i, i + 1) for i in range(10) if i % 2 == 0) # Don't do this
{i : i + 1 for i in range(10) if i % 2 == 0} # Use a dict comprehension instead!

# If you want a tuple:
tuple(i for i in range(10) if i % 2 == 0) # Unfortunately, there's no better way to do this
</source>
In general, generators and comprehensions should be used whenever possible - they're the fastest and easiest way to build collections. If you ever see yourself writing code like this:

<source lang="python">
# BAD PRACTICE! DO NOT USE!
digits = []
for digit in range(10):
digits.append(digit)
</source>
Don't do it! Instead, use a generator expression or comprehension, as appropriate. A good way to decide whether a generator expression or comprehension is better for this situation is that, if you're just going to be iterating over your collection once, you should use an generator expression. If not, use a comprehension.

=== Sorting ===

Sorting in Python is frequently performed using the <code>sorted</code> function, which takes two arguments: an iterable (anything you can do a for-loop over, including lists and tuples) and a function used to evaluate each item. If the function is omitted, Python will try to sort the lists by value.

The following example demonstrates using an inline function, which Python calls a ''lambda''.

<source lang="python">
fruits = ['apple', 'banana', 'cherry', 'date']

# sort the fruits by string length
new_fruits = sorted(fruits, key=lambda v: len(v) // 2)

print(new_fruits) # ['date', 'apple', 'banana', 'cherry']
</source>
Alternatively, you can sort a <code>list</code> using the <code>.sort</code> method. This is faster than calling <code>sorted</code>, but it modifies the list in-place and returns <code>None</code>.

<source lang="python">
fruits = ['apple', 'banana', 'cherry', 'date']
fruits.sort()
</source>
=== Import ===

If you want to use functions from other libraries (including ones that you install using pip), use <code>import</code>:

<source lang="python">
import time

current_time = time.localtime()
print(time.strftime('%a, %d %b %Y %H:%M:%S', current_time))
</source>
If you want to pull the functions out of their namespace, you can use <code>from ___ import ___</code> syntax:

<source lang="python">
from time import localtime, strftime

current_time = localtime()
print(strftime('%a, %d %b %Y %H:%M:%S', current_time))

# Be aware that this technique, although convenient, may cause unexpected behavior if the function names that you're pulling out of the namespace are already used for other purposes in Python.
</source>
=== File I/O ===

You can read an entire file into a variable like this:

<source lang="python">
f = open("example.txt")
file_contents = f.read()

f.close() # free up memory when we're finished with the file
</source>
A better option is to use a with-block, which handles opening and closing the file for you automatically:

<source lang="python">
with open("example.txt") as f:
file_contents = f.read()
</source>
You can read a file line-by-line like this:

<source lang="python">
with open("example.txt") as f:
for line in f:
print(f"Read line: {line.strip()}")
</source>
You can write to a file like this:

<source lang="python">
with open("example.txt", "w") as f:
f.write("Hello\nWorld\n")
</source>

=== Command-Line Arguments ===

Command line arguments are accessible in the variable <code>sys.argv</code>.

The following example shows a program that expects a filename as its argument, and it prints a usage message if the argument is not present ([https://stackoverflow.com/questions/983201/python-and-sys-argv source]).

<source lang="python">
import sys, os

if len(sys.argv) < 2:
sys.exit(f"Usage: {sys.argv[0]} filename")

filename = sys.argv[1]

if not os.path.exists(filename):
sys.exit(f"Error: File '{sys.argv[1]}' not found")
</source>

=== Object-Oriented Programming ===

You can define and use a class like this:

<source lang="python">
class Food:
# constructor:
def __init__(self, name):
self.name = name

def format_name(self):
return f"Gotta love to eat {self.name}"

class Fruit(Food):
def get_definition(self):
return f"{self.format_name()} (fruit)"

fruit = Fruit("Cherry")
print(fruit.format_name())
print(fruit.get_definition())
</source>
This is the same example as in [[PHP#Object-Oriented_Programming|the PHP guide]].

Some things to notice:

* Instance methods ''always'' take <code>self</code> as their first argument, followed by any number of additional parameters. This can be misleading for programmers familiar with other languages, because the number of arguments you feed to the method is actually ''one less than'' the number of declared parameters. Whenever you call a method on a class instance, that instance is implicitly fed into the explicitly-declared <code>self</code> parameter of the method.
*: Note that the <code>self</code> variable has the same purpose as the <code>this</code> variable in languages like PHP, Java, JavaScript, and C++.
*There is no need for a <code>new</code> keyword in Python.

= Footnotes =
# {{note|interpreted}}This is an oversimplification. If you're interested, here's a quick explanation of how: In CPython (the reference implementation of Python) programs are compiled to bytecode, an intermediate representation, which is then interpreted.
[[Category:Module 4]]

Python

2023-01-10T06:09:31Z

Zach: /* Pip */

Languages like Java and C++ have lots of rules regarding variable types, syntax, return values, and so on. These can be cumbersome when you are trying to write short, quick scripts to perform tasks. This is where a ''scripting language'' comes into play.

'''Python''' is a language well-suited to rapid prototype development. It is generally an ''interpreted language'', which means that, generally speaking, a Python program is converted line by line into instructions for the computer to execute as the program runs, instead of having a separate compilation step beforehand {{ref|interpreted}}. The syntax is clean, and it is usually clear at first glance what is going on when you write in Python.

= Installation =

Python may already be installed on your system. To see whether or not it is, enter the command

<source lang="bash">
$ python3 --version
</source>
If it tells you a version of Python '''greater than or equal to 3.6''' then you're good to go. If not, you need to do a quick package install to get it up and running. Using yum, it's just <code>sudo yum install python36</code>.

If you'd like to be able to just type <code>python</code> instead of <code>python3</code>, you can add the following line to the <code>~/.bashrc</code> file on your instance:

<source lang="bash">
alias python="/usr/bin/python3"
</source>

==== Python 2 vs Python 3 ====

In 2008, a new version of Python – Python 3 – was released, with a number of changes that made it incompatible with the previous version, Python 2. For several years, the Python community was slow to make the change to Python 3, but today, Python 3 is ''the'' standard for new code. The ''only'' reason to write Python 2 code today is because you have to work in an environment or with libraries that do not support Python 3, and that situation is becoming increasingly rare.

'''For CSE 330 (and this guide), you ''must'' use Python 3.6 or greater.'''

=== Pip ===

Linux distributions have package managers like ''Apt'', ''Yum'', and ''YaST''. PHP has a package manager named ''PEAR''. It's now time to introduce Python's leading package manager: '''pip'''.

Once you have pip installed, you can use it to install Python packages. Use the <code>pip-3.6</code> command:

<source lang="bash">
$ pip-3.6 install <package_name> # RHEL
</source>

If you'd like to be able to just type <code>pip</code> instead of <code>pip-3.6</code>, you can add the following line to the <code>~/.bashrc</code> file on your instance:

<source lang="bash">
alias pip="/usr/bin/pip-3.6"
</source>

= Running Python =

There are two common ways to run Python code: via the console, and via a Python script file.

=== The Python Console ===

The Python console enables you to experiment with code without opening a text editor. To enter the Python console, simply type the ''python'' command at the terminal:

<source lang="bash">
$ python3
</source>
To leave the interactive console, either type <code>quit()</code> or press <code>Ctrl-D</code> (on both Mac and Windows).

=== Python Script Files ===

You can also save Python script files for later use. The extension for Python scripts is <code>*.py</code>. To run a script file, simply feed its path as an argument to the <code>python</code> command in Terminal:

<source lang="bash">
$ python3 my_script.py
</source>

= Python Syntax and Language Components =

This section contains a very brief overview of Python syntax. For a more comprehensive introduction, see [http://docs.python.org/tutorial/ the Python docs].

=== Python's Type System ===

Python is a dynamically-typed, strongly-typed, and duck-typed language. What does that mean? Well, let's take those categories one by one:

<ol>
<li>
'''Dynamic typing''', in contrast to ''static typing'' means that the types of objects can be changed. In Java, for instance, if you tried to write the following code:

<source lang="java">
int c = 2;
c = "fish";
</source>

Your code would fail to compile, because the Java compiler wouldn't allow you to change the type of the variable <code>c</code> from <code>int</code> to <code>String</code>.

Python, however, is just fine with that. You can write:

<source lang="python">
c = 2
c = "fish"
</source>

And Python will happily do what you ask.
</li>

<li>
'''Strong typing''', in contrast to ''weak typing'', means, roughly, that a language won't implicitly convert types for you. So far, you've used a lot of PHP, which is very weakly typed. For example:

<source lang="php">
echo "10" . 2;
</source>

In PHP, this works just fine. The PHP interpreter sees that <code>2</code> is an int, and silently converts it to a string for you, so that <code>"10". 2</code> equals <code>"102"</code>. The problem with weak typing is that it can make it unclear to the programmer why things are happening the way they are. For instance, in PHP, the expression <code>"10" == 10</code> is true, which can be less than obvious. Python, however, tries to disallow that sort of thing:

<source lang="python">
"10" + "10" # Allowed – concatenates the two strings.
10 + 10 # Allowed – adds the two ints.
"10" + 10 # Not allowed – raises a TypeError
</source>

Perhaps the one exception to this rule is the relationship between numeric types; it's perfectly fine to add a float to an int, for example.
</li>

<li>
'''Duck-typing''' is a little bit different than the other categories, because it doesn't really have a direct opposite. Essentially, duck-typing is the philosophy that "If it walks like a duck, and quacks like a duck, it's a duck." In programming terms, that means that we often don't care what type an object is, as long as it can do the things we ask.

Let's look at an example:

<source lang="python">
# This function doubles the value it's passed.
def double(thing):
return thing + thing
double(10) # Returns 20
double("fish") # Returns "fishfish";
</source>

If this were Java, we'd have to write two different methods here: one for things of type <code>int</code>, and one for things of type <code>String</code>. Since this is Python, though, we don't have to do that. Instead, Python lets us say, essentially: "This function uses the <code>+</code> operator, so you can pass whatever object you want to the function, as long as the function supports the <code>+</code> operator".

To quote [https://groups.google.com/forum/?hl=en#!msg/comp.lang.python/CCs2oJdyuzc/NYjla5HKMOIJ Alex Martelli], one of the key people in the Python project,

<blockquote>
"[Don't] check whether it IS-a duck: check whether it QUACKS-like-a duck, WALKS-like-a duck, etc, etc, depending on exactly what subset of duck-like behaviour you need."
</blockquote>

</li>
</ol>

=== Basic Types ===

In Python, the basic scalar types are:

# <code>str</code> – a string.
# <code>int</code> – an integer object with unlimited digits.
# <code>bool</code> – a Boolean value.
# <code>float</code> – a floating point number; generally equivalent to <code>double</code> in Java.
# <code>bytes</code> – represents a collection of bytes; you're unlikely to use this often.

Casting is done in the C style - to parse an <code>int</code> from a string, for instance, you'd write <code>int("123")</code>.

=== Basic Control Flow ===
In Python, blocks of code are delineated using colons and indentation instead of curly braces. '''''That means that proper indentation is mandatory.''''' Every time you use a colon, you must indent one step further, and every time a block ends, you must unindent a step.

Conditional statements are done using the <code>if</code>, <code>elif</code>, and <code>else</code> operators, like in the example below - pay attention to the indentation:

<source lang="python">
cookies_left = 10
cake_slices_left = 3
if cookies_left > 10:
print("Lots of cookies!")
elif cookies_left > 8:
print("Plenty left!")
elif cookies_left == 5:
print("Halfway done!")
else:
if cake_slices_left == 3:
print("Lots of cake!")
elif cake_slices_left == 2:
print("One slice gone!")
else:
print("Little to no cake left!")
</source>

=== An Example Python Script ===

In my mind, there's no better way to learn Python than to be immersed in a simple example script.

<source lang="python">
print("Hello World")

fruits = ["apple", "banana", "cherry", "date"]
for fruit in fruits:
print(f"I always love to eat a fresh {fruit}")

# Map the fruits list over to a new list containing the length of the fruit strings:
fruit_size = [len(fruit) for fruit in fruits]

avg_fruit_size = sum(fruit_size) / len(fruit_size)
print(f"The average fruit string length is {avg_fruit_size:.2f}.")
</source>

Some things to notice:

* Printing is achieved using the <code>print()</code> function.
* Comments start with a pound symbol <code>#</code>.
* We can transform/map a list to a new list in just one line!

For some more examples, see [https://wiki.python.org/moin/SimplePrograms the Python wiki].

=== A Few Gotchas ===

Here are a few things you may not expect when starting out with Python:

<ol>

<li>
Strings can be delimited using either single quotes (<code>'</code>) or double quotes (<code>"</code>), so <code>'Python'</code> is the same as <code>"Python"</code>. You can also use three sets of quotes (single or double again) to delineate a multiple-line string, like so:
<source lang="python">
long_string = """This
is my
very long
multiple-line
string"""
</source>
</li>

<li>
To compare the equality of two objects, use the <code>==</code> operator. To compare the identity of two objects (i.e. whether or not they are literally the same object, at the same place in memory), use the <code>is</code> operator. These correspond, respectively to <code>.equals()</code> and <code>==</code> in Java.
</li>

<li>
The <code>/</code> operator performs floating-point division, not integer division, even when both its arguments are integers. If you want to do integer division, use the <code>//</code> operator. Let's look at an example:

<source lang="python">
9 / 10 == 0.9 # Floating-point division
9.0 / 10.0 == 0.9 # Floating-point division
9 // 10 == 0 # Integer division
9.0 // 10.0 == 0.0 # Integer division, even though the resulting value is a float
</source>
</li>

<li>
As discussed in the section [[#Python's_Type_System|Python's Type System]], you can't combine strings and numbers using the <code>+</code> operator. Instead, you'll have to use one of the following methods of string formatting:

<source lang="python">

# Technique 1 – printf-style Formatting
# This is a classic style of string formatting, used in many programming languages, including in Java's String.format.
"You scored %d out of %d!" % (score, questions)

# Technique 2 – str.format
"You scored {} out of {}!".format(score, questions)

# Technique 3 – f-strings
# These were only introduced in Python 3.6 but they're very convenient.
f"You scored {score} out of {questions}!"

# Technique 4 - String Concatenation
# This is fine for small things, but for more complicated stuff, use one of the other techniques.
"You scored " + str(score) + " out of " + str(questions) + "!"
</source>
</li>
</ol>

=== Functions ===

Define functions using the <code>def</code> keyword:

<source lang="python">
def hello(name):
print(f"Hello, {name}!")

</source>

Invoke functions using parentheses:
<source lang="python">
hello("Batman")
hello("Superman")
</source>

Functions can have optional arguments; however, these must come after any required arguments:
<source lang="python">
def hello(name="User"):
print(f"Hello, {name}!")

hello() # Prints "Hello, User!"
hello("Sarah") # Prints "Hello, Sarah!"
</source>

Functions are just regular objects in Python, meaning you can do stuff like this:

<source lang="python">
def hello(name="User"):
print(f"Hello, {name}!")

greeting = hello

greeting("Siwei") # Prints "Hello, Siwei!"
</source>

=== Collection Types ===
Python has a few basic, built-in collection types. They each have their own special use cases, but they share a lot of similarities. All built-in data structures in Python are zero-indexed, for instance.

To obtain the number of elements in a collection, use the <code>len()</code> function, like this:

<source lang="python">
pets = ['Dog', 'Cat', 'Fish'] # This is a list; you'll learn about them below.

len(pets) # Returns 3
</source>

To check if a collection contains an item, use the <code>in</code> operator:
<source lang="python">
pets = ['Dog', 'Cat', 'Fish'] # This is a list; you'll learn about them below.

'Dog' in pets # Returns True
</source>

Helpfully, the Python Software Foundation maintains a page containing the asymptotic complexities of important operations on major Python data structures. It's available [https://wiki.python.org/moin/TimeComplexity here].

==== Lists ====

Unlike languages like Java and C, Python's array-like type doesn't have fixed length. Instead, a <code>list</code> is a dynamic array (like Java's <code>ArrayList</code>) of objects of any type. Here's an example:

<source lang="python">
fruits = ["Apple", "Pear", "Orange"] # Lists can be creating by placing their items between square brackets

random_items = ['Apple', 12312, 2.0, [1, 2, 3]] # Lists can contain objects of different types - even other lists!

pets.append('Turtle') # Adds 'Turtle' to the end of pets
</source>

For the next few examples, we'll be using the following list:

<source lang="python">
pets = ['Dog', 'Cat', 'Fish', 'Turtle', 'Lizard', 'Snake']
</source>

To access or assign elements of a list, use the <code>[]</code> operator.

<source lang="python">
pets[0] # Returns 'Dog'
pets[1] = 'Cow' # Sets pets[1] to 'Cow'
pets[1] = 'Cat' # Sets pets[1] back to 'Cat'
</source>

You can also access or assign lists using negative indices, which will return elements starting from the end of the list:

<source lang="python">
pets[-1] # Returns 'Snake'
pets[-2] # Returns 'Lizard'
</source>

You can also use 'slice notation' to obtain or assign subsets of lists:
<source lang="python">
pets[start:end] # Returns items from index start to index end - 1.
pets[start:] # Returns items from index start to the end of the list, inclusive.
pets[:end] # Returns items from the beginning of the list to index end - 1.
pets[:] # Returns every item in the list
</source>

There's also a third, optional argument: <code>step</code>, which lets you jump over values:
<source lang="python">
pets[start:end:step] # Returns items from index start to index end - 1, jumping step values at a time

# Examples:
pets[1:5:2] # Returns items from index 1 to index 4, jumping 2 at a time, so ['Cat', 'Turtle']
pets[::3] # Returns all items in the list, jumping 3 at a time, so ['Dog', 'Turtle']
pets[::-1] # Returns all items in the list, jumping -1 at a time. This reverses the list, so ['Snake', 'Lizard', 'Turtle', 'Fish', 'Cat', 'Dog']
</source>

Lists have all the performance characteristics you'd expect of a dynamic array - constant-time indexing, amortized constant-time appends, and so on.

==== Tuples ====

Python has a special data type called <code>tuple</code>, which is exactly the same as a list except for the fact that it cannot be modified, so you can't append or replace items.

<source lang="python">
cities = ('St. Louis', 'Los Angeles', 'Seattle') # Tuples are defined using parentheses.
numbers = 1, 2, 3 # You can also omit the parentheses if it's unambiguous
single_item_tuple = (1,) # To make a tuple with only one item, put a comma after it
</source>

Tuples also enable you to have multiple return values from a function:

<source lang="python">
def compute_length(string):
str_len = len(string)
if str_len < 5:
return str_len, "short"
elif str_len < 40:
return str_len, "medium"
return str_len, "long"

length, description = compute_length("Four score and seven years ago")
print(f"The {description} string is {length} characters long.")
</source>
The above example also demonstrates Python's if...elif...else conditional structure.

==== Dictionaries ====

Python has another datatype called a '''dictionary''' (or <code>dict</code>, for short), which are like ''Maps'' in Java, ''associative arrays'' in PHP, and ''object literals'' in JavaScript (coming up soon in Module 6). Essentially, they enable you to use any immutable object as the key in your data structure. <code>dict</code> objects can be created using either the <code>dict</code> function, or by placing items between curly braces, separating keys and values using colons:

<source lang="python">
fruits_in_bowl = {
'apple': 4,
'banana': 2,
'cherry': 0,
'date': 12
}

apple_count = fruits_in_bowl['apple'] # Equals 4

fruits_in_bowl['pear'] = 9 # Sets 'pear' in the dictionary to 9.

for fruit, num in fruits_in_bowl.items():
print(f"There are {num} {fruit}(s) in the bowl." % (num, fruit))
</source>
If you only want the keys from a dictionary, you can just iterate over it directly, like this:

<source lang="python">
for fruit in fruits_in_bowl:
print(fruit)
</source>
If you want only the values but not the keys from a dictionary, use <code>my_dictionary.values()</code>.

==== Sets ====
A <code>set</code> is an unordered collection of unique items, with constant-time membership checking.
<source lang="python">
fruits = {'Apple', 'Apple', 'Apple', 'Pear'} # The set removes duplicates, so it will only contain {'Apple', 'Pear'}
fruits = set(['Apple', 'Apple', 'Apple', 'Pear']) # This line transforms a list into a set
print('Apple' in fruits) # Constant-time!
</source>

=== Loops ===

In Python, while loops are written like this:

<source lang="python">
i = 0
while i < 10:
i += 1
</source>
For loops are written like this:

<source lang="python">
pets = ['Dog', 'Cat', 'Fish']

for pet in pets:
print(pet)
</source>
To iterate over a range of numbers, you can use the convenient <code>range</code> function, like so:

<source lang="python">
for i in range(20, 30): # If you omit the first argument to range, it's assumed to be zero.
print(i)
</source>
Note that <code>range</code> provides a closed-open range. That is to say, it is inclusive of its lower bound and exclusive of its upper bound. To be very explicit about it:

<source lang="python">
list(range(3, 6)) == [3, 4, 5]
</source>
Unlike in other languages you may know, the following is not good practice in Python:

<source lang="python">
pets = ['Dog', 'Cat', 'Fish']

# BAD PRACTICE. DO NOT USE.
for i in range(len(pets)):
pet = pets[i]
</source>
Instead, if you need to access both a variable and its index, use the <code>enumerate</code> function, which allows you to iterate through a group of items, along with their indexes, like so:

<source lang="python">
pets = ['Dog', 'Cat', 'Fish']

for index, pet in enumerate(pets):
print(f"{pet} at index {index}")
</source>

=== Generator Expressions and Comprehensions ===

A generator is a special kind of object that, basically, represents a collection of items using a value and a function for generating the next value in the collection. In other words, it's a way of creating something that behaves like a list, but which only generates one value at a time, and discards a value when it's done with it. Say, for example, you wanted to do an operation on 10,000,000,000,000 numbers. You could put them all in a list, then iterate over the list, but that would take a lot of memory, plus you'd have to create and fill the entire list before beginning to actually work on your problem. Instead, you'd probably be better off using a generator; telling the computer, essentially: “The starting value is 0. When I ask for the next number in the collection, add 1 to current value and give me that.”

A generator expression is an expression that is evaluated to create a generator. They're super useful. Generator expressions can take two forms – they can be used with and without a condition:

<source lang="python">
(EXPRESSION for VARIABLE in ITERABLE) # Unconditional
(EXPRESSION for VARIABLE in ITERABLE if CONDITION) # Conditional
</source>
For our purposes, you can think of an iterable as anything you can do a for loop over. Let's take a look at a generator expression in use:

<source lang="python">
double_even_digits = (number * 2 for number in range(10) if number % 2 == 0)
</source>
That line says that <code>double_even_digits</code> is every even number between 0 and 10 multiplied by two. Simple enough, right? If we wanted to use all of the numbers, not just the even ones, we could simply remove the if statement at the end of the expression.

Here's the thing, though: if you were to print <code>double_even_digits</code>, instead of a nice bunch of numbers, you'd get something that looks like this:

<code><generator object <genexpr><code>at 0x107fc92b0></code>

That's because generators are lazy! They don't generate their values until those values are needed, and they discard values once they're no longer needed. As a result, '''you can only iterate over a generator once'''. In this case, you can't print out the whole generator, because it hasn't generated all the values yet. You can, however, iterate over the generator, and print its values one at a time, like so:

<source lang="python">
for number in double_even_digits:
print(number)
</source>
You can also make a list out of the generator, which will go through all the items in the generator and put them in a list, like so:

<source lang="python">
even_number_list = list(double_even_digits)
</source>
That's actually a great segue to the second topic of this section: comprehensions. A comprehension is just a prettier way to make lists, sets, and dicts out of a generator expressions.

<source lang="python">
# If you want a list:
list(i for i in range(10) if i % 2 == 0) # Don't do this
[i for i in range(10) if i % 2 == 0] # Use a list comprehension instead!

# If you want a set:
set(i for i in range(10) if i % 2 == 0) # Don't do this
{i for i in range(10) if i % 2 == 0} # Use a set comprehension instead!

# If you want a dict:
dict((i, i + 1) for i in range(10) if i % 2 == 0) # Don't do this
{i : i + 1 for i in range(10) if i % 2 == 0} # Use a dict comprehension instead!

# If you want a tuple:
tuple(i for i in range(10) if i % 2 == 0) # Unfortunately, there's no better way to do this
</source>
In general, generators and comprehensions should be used whenever possible - they're the fastest and easiest way to build collections. If you ever see yourself writing code like this:

<source lang="python">
# BAD PRACTICE! DO NOT USE!
digits = []
for digit in range(10):
digits.append(digit)
</source>
Don't do it! Instead, use a generator expression or comprehension, as appropriate. A good way to decide whether a generator expression or comprehension is better for this situation is that, if you're just going to be iterating over your collection once, you should use an generator expression. If not, use a comprehension.

=== Sorting ===

Sorting in Python is frequently performed using the <code>sorted</code> function, which takes two arguments: an iterable (anything you can do a for-loop over, including lists and tuples) and a function used to evaluate each item. If the function is omitted, Python will try to sort the lists by value.

The following example demonstrates using an inline function, which Python calls a ''lambda''.

<source lang="python">
fruits = ['apple', 'banana', 'cherry', 'date']

# sort the fruits by string length
new_fruits = sorted(fruits, key=lambda v: len(v) // 2)

print(new_fruits) # ['date', 'apple', 'banana', 'cherry']
</source>
Alternatively, you can sort a <code>list</code> using the <code>.sort</code> method. This is faster than calling <code>sorted</code>, but it modifies the list in-place and returns <code>None</code>.

<source lang="python">
fruits = ['apple', 'banana', 'cherry', 'date']
fruits.sort()
</source>
=== Import ===

If you want to use functions from other libraries (including ones that you install using pip), use <code>import</code>:

<source lang="python">
import time

current_time = time.localtime()
print(time.strftime('%a, %d %b %Y %H:%M:%S', current_time))
</source>
If you want to pull the functions out of their namespace, you can use <code>from ___ import ___</code> syntax:

<source lang="python">
from time import localtime, strftime

current_time = localtime()
print(strftime('%a, %d %b %Y %H:%M:%S', current_time))

# Be aware that this technique, although convenient, may cause unexpected behavior if the function names that you're pulling out of the namespace are already used for other purposes in Python.
</source>
=== File I/O ===

You can read an entire file into a variable like this:

<source lang="python">
f = open("example.txt")
file_contents = f.read()

f.close() # free up memory when we're finished with the file
</source>
A better option is to use a with-block, which handles opening and closing the file for you automatically:

<source lang="python">
with open("example.txt") as f:
file_contents = f.read()
</source>
You can read a file line-by-line like this:

<source lang="python">
with open("example.txt") as f:
for line in f:
print(f"Read line: {line.strip()}")
</source>
You can write to a file like this:

<source lang="python">
with open("example.txt", "w") as f:
f.write("Hello\nWorld\n")
</source>

=== Command-Line Arguments ===

Command line arguments are accessible in the variable <code>sys.argv</code>.

The following example shows a program that expects a filename as its argument, and it prints a usage message if the argument is not present ([https://stackoverflow.com/questions/983201/python-and-sys-argv source]).

<source lang="python">
import sys, os

if len(sys.argv) < 2:
sys.exit(f"Usage: {sys.argv[0]} filename")

filename = sys.argv[1]

if not os.path.exists(filename):
sys.exit(f"Error: File '{sys.argv[1]}' not found")
</source>

=== Object-Oriented Programming ===

You can define and use a class like this:

<source lang="python">
class Food:
# constructor:
def __init__(self, name):
self.name = name

def format_name(self):
return f"Gotta love to eat {self.name}"

class Fruit(Food):
def get_definition(self):
return f"{self.format_name()} (fruit)"

fruit = Fruit("Cherry")
print(fruit.format_name())
print(fruit.get_definition())
</source>
This is the same example as in [[PHP#Object-Oriented_Programming|the PHP guide]].

Some things to notice:

* Instance methods ''always'' take <code>self</code> as their first argument, followed by any number of additional parameters. This can be misleading for programmers familiar with other languages, because the number of arguments you feed to the method is actually ''one less than'' the number of declared parameters. Whenever you call a method on a class instance, that instance is implicitly fed into the explicitly-declared <code>self</code> parameter of the method.
*: Note that the <code>self</code> variable has the same purpose as the <code>this</code> variable in languages like PHP, Java, JavaScript, and C++.
*There is no need for a <code>new</code> keyword in Python.

= Footnotes =
# {{note|interpreted}}This is an oversimplification. If you're interested, here's a quick explanation of how: In CPython (the reference implementation of Python) programs are compiled to bytecode, an intermediate representation, which is then interpreted.
[[Category:Module 4]]

MySQL Schema and State

2023-01-10T06:06:12Z

Zach: /* Unique Key */

MySQL is a powerful database that allows for complex logic when manipulating data. This guide is an introduction and a reference to common SQL queries for schema and state manipulation.

This guide assumes that you already have a working copy of MySQL. If you don't, read [[Introduction to MySQL]] first.

==Schema ==

The ''schema'' of your database refers to the way that your data is structured and to how bits of data relate to each other. Every database in your MySQL server has a separate schema, and separate databases should not typically interact with one another.

=== Tables ===

A database contains one or more ''tables''. Tables contain columns and rows, which you may sometimes call ''fields'' and ''entries''.

You can see all of the tables in the database using this query:

<source lang="mysql">
show tables;
</source>

When you create a table, you should provide information about all of that table's columns within the ''create table'' query. Here is the general syntax for the query:

<source lang="mysql">
create table <TABLENAME> (
<FIELDNAME> <DATATYPE> <OPTIONS>,
<FIELDNAME> <DATATYPE> <OPTIONS>,
<FIELDNAME> <DATATYPE> <OPTIONS>,
<FIELDNAME> <DATATYPE> <OPTIONS>
) <TABLE_OPTIONS>;
</source>

You can get details about an existing table using this query:

<source lang="mysql">
describe <TABLENAME>;
</source>

==== Data Types ====

Just like a programming language, MySQL has data types. There are dozens of data types; some of the most common ones are documented below.

===== String Data Types =====

There are four main types of string data types: character, text, blob, and enumerable.

* Character types are useful for short strings, like usernames and e-mail addresses.
* Text types differ from character types in that behind the scenes, they are not actually stored with the rest of the table row; they are useful for longer blocks of text, like forum posts.
* Blobs are for binary data, like files.
* Enumerables are for something that could have one of a set number of values (not more than 65535), like what State a user is from (Alabama, Alaska, and so on). Enums are also useful for storing boolean data, like '''ENUM('yes','no')'''. An enum is actually stored as an integer in the database, making queries more efficient.

The syntax for these data types is:

* '''VARCHAR(#)''', a string ''at most'' # characters long, where 0 ≤ # ≤ 65535
* '''CHAR(#)''', a string ''exactly'' # characters long, where 0 ≤ # ≤ 65535
* '''TINYTEXT''', a text block at most 255 characters long
* '''TEXT''', a text block at most 65535 characters long
* '''LONGTEXT''', a text block at most 4 GB in length
* '''BLOB''', a binary string at most 65 KB in length
* '''MEDIUMBLOB''', a binary string at most 16 MB in length
* '''LONGBLOB''', a binary string at most 4 GB in length
* '''ENUM('a', 'b', 'c')''', an enumerable with choices a, b, and c

===== Numeric Data Types =====

There are two types of numeric data types that you need to worry about: integers and decimals. The difference between the two should be pretty self-explanatory. The syntax for using these data types is:

* '''TINYINT''', an integer ''n'' where -128 ≤ n ≤ 127
* '''TINYINT UNSIGNED''', an integer ''n'' where 0 ≤ n ≤ 255
* '''SMALLINT''', an integer ''n'' where -32,768 ≤ n ≤ 32,767
* '''SMALLINT UNSIGNED''', an integer ''n'' where 0 ≤ n ≤ 65,535
* '''MEDIUMINT''', an integer ''n'' where -8,388,608 ≤ n ≤ 8,388,607
* '''MEDIUMINT UNSIGNED''', an integer ''n'' where 0 ≤ n ≤ 16,777,215
* '''INT''', an integer ''n'' where -2,147,483,648 ≤ n ≤ 2,147,483,647
* '''INT UNSIGNED''', an integer ''n'' where 0 ≤ n ≤ 4,294,967,295
* '''BIGINT''', an integer ''n'' where -9,223,372,036,854,775,808 ≤ n ≤ 9,223,372,036,854,775,807
* '''BIGINT UNSIGNED''', an integer ''n'' where 0 ≤ n ≤ 18,446,744,073,709,551,615
* '''DECIMAL(#a, #b)''', a decimal number at most ''#a'' digits long, ''#b'' of which are after the decimal point. For example, ''DECIMAL(5,2)'' can contain ''n'' where -999.99 ≤ n ≤ 999.99

When choosing the size of an integer, you should use the ''smallest integer for which you know you will never "run out of room"''. In most everyday circumstances, '''MEDIUMINT''' should be the biggest you'll need.

===== Temporal Data Types =====
{{XKCD
|name=2038
|id=607
}}

MySQL provides a class of data types that specialize in handling dates and times. The syntax for using date types is:

* '''DATE''', a date ''d'' where <code>1000-01-01</code> ≤ d ≤ <code>9999-12-31</code>
* '''DATETIME''', a date with time ''d'' where <code>1000-01-01 00:00:00</code> ≤ d ≤ <code>9999-12-31 23:59:59</code>
* '''TIMESTAMP''', a date with time ''d'' where <code>1970-01-01 00:00:01 UTC</code> ≤ d ≤ <code>2038-01-19 03:14:07 UTC</code> (note: <code>1970-01-01 00:00:01 UTC</code> is the Unix epoch)
* '''TIME''', a length of time ''t'' where <code>-838:59:59</code> ≤ t ≤ <code>838:59:59</code> (note: you cannot store time to a precision greater than 1 second)
* '''YEAR''', a year ''y'' where 1901 ≤ y ≤ 2155

Note that '''TIMESTAMP''' is a highly versatile data type that can automatically change whenever a row is updated. For more information, refer to [http://dev.mysql.com/doc/refman/5.6/en/timestamp-initialization.html the MySQL documentation].

==== Field Options ====

By default, any "cell" in a MySQL table can contain the ''NULL'' value. If a certain field in your database schema will never contain a null value, you should declare it as '''NOT NULL''' in order to make queries more efficient.

You can also specify a default value for a field. If you do, when you insert data into the table, it won't be necessary to specify a value for that column. To specify a default value for a field, use the syntax '''DEFAULT 'DEFAULT-VALUE' '''

==== Create Table Example ====

The following query would create a table called ''employees'' with six columns: employee ID, first name, last name, nickname, department, and the date that they joined the company. Nickname is the only optional column. The primary key for the table (see the next section) will be the ''id'' column. The department will be one of five choices, with the default being CSE. The storage engine for the table will be InnoDB, and the default charset for the table will be UTF8.

<source lang="mysql">
create table employees (
id mediumint unsigned not null auto_increment,
first_name varchar(30) not null,
last_name varchar(40) not null,
nickname varchar(20),
department enum('CSE','BME','EECE','ESE','MEMS') not null default 'CSE',
joined timestamp not null default current_timestamp,
primary key (id)
) engine = InnoDB default character set = utf8 collate = utf8_general_ci;
</source>

=== Keys ===

When MySQL goes to get data out of a database, it does so using an index, like the index of a textbook. In tech talk, we call indices ''keys''.

==== Primary Key ====

The '''primary key''' is what MySQL uses to distinguish one row from another. Every table must have exactly one primary key; no more, no less.

It is common practice to add a column to a table with data type '''mediumint unsigned not null auto_increment''' to serve as the primary key for a table. The '''auto_increment''' is a neat little feature of MySQL that basically says, "when a new row is inserted into this table, assign it the next-lowest available ID".

To specify a primary key for a table, use the syntax:

<source lang="mysql">
primary key (column1, column2, column3)
</source>

==== Unique Key ====

If you want to specify that all items in a certain column will be unique from each other (like usernames or e-mail addresses), you can use a '''unique key'''.

Because a table can have more than one unique key, you need to give your unique key a name. It is common practice to name the key after the columns it is indexing, like ''idx_unique_column1_column2''.

The syntax for specifying a unique key is:

<source lang="mysql">
unique key INDEX_NAME (column1, column2, column3)
</source>

==== Foreign Key ====

A '''foreign key''' is the most complicated type of MySQL index. A foreign key says that the values in one column or columns are associated with the values in a column or columns in a ''different table''.

For example, suppose you had two tables, one containing users and the other containing posts on a forum. There is a one-to-many relationship between users and posts. You could link a post to a user by having a column in your post table called ''user_id'', which would be a ''foreign key'' to the ''id'' column in your users table.

Foreign keys can be a source of great confusion in developing database schemas. If you get stuck, keep this tip in mind: '''Foreign Keys should always reference the columns used in the Primary Key of a different table.'''

The syntax for creating a foreign key is as follows:

<source lang="mysql">
foreign key (column1, column2, column3) references FOREIGN_TABLENAME (column1, column2, column3)
</source>

For example, in the user-and-post example above, the foreign key declaration would be:

<source lang="mysql">
foreign key (user_id) references user (id)
</source>

===== Further Reading about Foreign Keys =====

* [[wikipedia:Foreign Key|Wikipedia Entry on Foreign Keys]]
* [http://www.techrepublic.com/article/an-introduction-to-foreign-keys-and-referential-integrity-in-mysql/6035435 An introductory article on Tech Republic]
* [http://dev.mysql.com/doc/refman/5.1/en/innodb-foreign-key-constraints.html MySQL Documentation for Foreign Keys]

==== Syntax for Keys ====

You can define keys in your '''create table''' query, or you can define them later.

If you define them in your '''create table''' query, do so after the column definitions but still inside the parentheses:

<source lang="mysql">
create table TABLENAME (
column,
column,
....,
primary key (column1, column2, ...),
unique key INDEXNAME (column1, column2, ...),
foreign key (column1, column2, ...) references FOREIGN_TABLENAME (column1, column2, ...)
)
</source>

To define indices later, use an '''ALTER TABLE''' query:

<source lang="mysql">
alter table TABLENAME add unique key INDEXNAME (column1, column2, ...);
alter table TABLENAME add foreign key (column1, column2, ...) references FOREIGN_TABLENAME (column1, column2, ...);
</source>

== State ==

The ''state'' of your database refers to the data that it holds. Whenever you insert, update, or delete data, the state of your database changes.

=== Inserting Data from a Query ===

To insert a row (aka ''entry'') into a table, use a query of the form

<source lang="mysql">
insert into TABLENAME (column1, column2, column3) values (value1_1, value2_1, value3_1), (value1_2, value2_2, value3_2), ...
</source>

For example, to insert an employee into the table we created earlier, you could use the query

<source lang="mysql">
insert into employees (first_name, last_name, department) values ('John', 'Doe', 'BME'), ('Keith', 'Jones', 'ESE');
</source>

Because of the various default and auto_increment values we have specified in our database, the database would now contain the following data:

+----+------------+-----------+----------+------------+---------------------+
| id | first_name | last_name | nickname | department | joined |
+----+------------+-----------+----------+------------+---------------------+
| 1 | John | Doe | NULL | BME | 2012-08-22 22:02:24 |
| 2 | Keith | Jones | NULL | ESE | 2012-08-22 22:02:24 |
+----+------------+-----------+----------+------------+---------------------+

If you want to insert a DATE or DATETIME value into a database, use the YYYY-MM-DD HH:MM:SS format. The exception is with TIMESTAMP, in which you should use a 16-bit integer representing the number of seconds since January 1, 1970.

=== Inserting Data from a File ===

The '''load data infile''' command populates a table using data stored in a text file. The general form of the command is:

<source lang="mysql">
LOAD DATA INFILE 'dataFile.txt' INTO TABLE table_name;
</source>

where ''dataFile.txt'' is located relative to the server's data directory if a path name with leading components is given; otherwise, if no leading components are given, the server assumes the file is in the default database's database directory. If an absolute path is specified, the server uses it. For this command to work, you must issue the following command to grant the FILE privilege to your user:

<source lang="mysql">
GRANT FILE on *.* to 'user'@'host';
</source>

Alternatively, and quite usefully, you can load data from a local file (a file residing on the same machine as the client) using the LOCAL keyword. When loading data from the local machine you do not need to grant your user the FILE privilege, think about why this might be the case. (Hint: what is going on behind the scenes when you load the data in from a local machine when your database is hosted on a different machine?)

<source lang="mysql">
LOAD DATA LOCAL INFILE '/home/linus_torvalds/Documents/dataFile.txt' INTO TABLE table_name;
</source>

'''Note:''' When the local keyword is used, it is best to specify an absolute path.

'''Additional Note:''' If we SSH into our EC2 instance and then run the mysql command, the client is the EC2 instance, not the local computer that we SSH from. The LOCAL keyword in this case is allowing us to load files from the EC2 instance, not the local computer that we are using.

By default, the text files are assumed to have tab-delimited columns with column values specified in the same order as they are defined in the table (though you can specify additional/alternative delimiters using additional options).

=== Updating Data ===

To update a row in a table, use a query of the form

<source lang="mysql">
update TABLENAME set column1=value1, column2=value2, column3=value3 where CONDITION
</source>

For example, to set Keith Jones's nickname to "KJ", you would run the query

<source lang="mysql">
update employees set nickname='KJ' where id=2;
</source>

=== Deleting Data ===

To delete a row from a table, use a query of the form

<source lang="mysql">
delete from TABLENAME where CONDITION
</source>

For example, to delete John Doe from the table, you would run the query

<source lang="mysql">
delete from employees where id=1;
</source>

'''DO NOT FORGET THE WHERE CLAUSE!''' If you do, ''all'' rows will be deleted from your table!

== Selecting Data ==

The most simple query to select data from a table is:

<source lang="mysql">
select * from TABLENAME;
</source>

However, in practice, you will almost always want to perform a more advanced query.

=== Selecting Specific Columns ===

In order to save transmission time, you should select only those columns that you need in your application. The syntax for selecting only certain columns is

<source lang="mysql">
select column1, column2, column3 from TABLENAME;
</source>

=== Computation ===

MySQL can perform computation. For example, if you wanted to select the amount of time that a certain employee has been at the company, you could run the query

<source lang="mysql">
select TIMESTAMPDIFF( second, joined, NOW() ) as time_with_company from employees;
</source>

=== Joins ===

When you have one-to-one, one-to-many, or many-to-many relationships between tables, you will often want to select data from multiple tables in one query. To do this, use '''joins'''.

Joins are best described by example. Suppose you had the following tables in a MySQL database:

mysql> describe employees;
+------------+---------------------------------------+------+-----+-------------------+----------------+
| Field | Type | Null | Key | Default | Extra |
+------------+---------------------------------------+------+-----+-------------------+----------------+
| id | mediumint(8) unsigned | NO | PRI | NULL | auto_increment |
| first_name | varchar(30) | NO | | NULL | |
| last_name | varchar(40) | NO | | NULL | |
| nickname | varchar(20) | YES | | NULL | |
| department | enum('CSE','BME','EECE','ESE','MEMS') | NO | | CSE | |
| joined | timestamp | NO | | CURRENT_TIMESTAMP | |
+------------+---------------------------------------+------+-----+-------------------+----------------+
6 rows in set (0.00 sec)

mysql> describe projects;
+--------------+-----------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+--------------+-----------------------+------+-----+---------+----------------+
| id | mediumint(8) unsigned | NO | PRI | NULL | auto_increment |
| employee_id | mediumint(8) unsigned | NO | MUL | NULL | |
| project_name | tinytext | NO | | NULL | |
+--------------+-----------------------+------+-----+---------+----------------+
3 rows in set (0.00 sec)

Further, suppose the tables had the following data:

mysql> select * from employees;
+----+------------+-----------+----------+------------+---------------------+
| id | first_name | last_name | nickname | department | joined |
+----+------------+-----------+----------+------------+---------------------+
| 1 | John | Doe | NULL | BME | 2012-08-22 22:02:24 |
| 2 | Keith | Jones | KK | ESE | 2012-08-22 22:02:24 |
+----+------------+-----------+----------+------------+---------------------+
2 rows in set (0.00 sec)

mysql> select * from projects;
+----+-------------+-----------------------------------+
| id | employee_id | project_name |
+----+-------------+-----------------------------------+
| 1 | 1 | Interview the Chancellor |
| 2 | 1 | Give Presentation in Mumbai |
| 3 | 2 | Publish Research Paper in Journal |
+----+-------------+-----------------------------------+
3 rows in set (0.00 sec)

If you ran the following query:

<source lang="mysql">
select
project_name,
employees.first_name,
employees.last_name
from projects
join employees on (projects.employee_id=employees.id);
</source>

You would get the result:

mysql> select project_name, employees.first_name, employees.last_name from projects
-> join employees on (projects.employee_id=employees.id);
+-----------------------------------+------------+-----------+
| project_name | first_name | last_name |
+-----------------------------------+------------+-----------+
| Interview the Chancellor | John | Doe |
| Give Presentation in Mumbai | John | Doe |
| Publish Research Paper in Journal | Keith | Jones |
+-----------------------------------+------------+-----------+
3 rows in set (0.00 sec)

Isn't that cool?

==== Inner Joins, Left Joins, Right Joins, and Full Joins ====

When you perform a '''JOIN''' like we did above, you are implicitly performing an '''INNER JOIN'''. However, there may be times when you want to select data in a query even if it doesn't have a complement in the joined table. For example, suppose Keith Jones cancelled his project. The projects table now contains the following data:

mysql> select * from projects;
+----+-------------+-----------------------------+
| id | employee_id | project_name |
+----+-------------+-----------------------------+
| 1 | 1 | Interview the Chancellor |
| 2 | 1 | Give Presentation in Mumbai |
+----+-------------+-----------------------------+
2 rows in set (0.00 sec)

Now, we could run the following query:

<source lang="mysql">
select
count(projects.id) as project_num,
employees.first_name,
employees.last_name
from employees
join projects on (employees.id=projects.employee_id)
group by employees.id;
</source>

We get the result:

mysql> select count(projects.id) as project_num, employees.first_name, employees.last_name from employees
-> join projects on (employees.id=projects.employee_id) group by employees.id;
+-------------+------------+-----------+
| project_num | first_name | last_name |
+-------------+------------+-----------+
| 2 | John | Doe |
+-------------+------------+-----------+
1 row in set (0.00 sec)

But where's Keith Jones? Since Keith Jones didn't have any projects, he was omitted from the result. This is probably not what we intended to happen. Instead, you should perform a '''LEFT JOIN''', like so:

<source lang="mysql">
select
count(projects.id) as project_num,
employees.first_name,
employees.last_name
from employees
left join projects on (employees.id=projects.employee_id)
group by employees.id;
</source>

The result will now be:

mysql> select count(projects.id) as project_num, employees.first_name, employees.last_name from employees
-> left join projects on (employees.id=projects.employee_id) group by employees.id;
+-------------+------------+-----------+
| project_num | first_name | last_name |
+-------------+------------+-----------+
| 2 | John | Doe |
| 0 | Keith | Jones |
+-------------+------------+-----------+
2 rows in set (0.00 sec)

(For more information on COUNT and GROUP BY, see [[#Aggregation Clauses|a later section]].)

You can see that a LEFT JOIN preserves all rows from the "left" table in the join. Likewise, a RIGHT JOIN preserves all rows from the "right" table. A FULL JOIN preserves all rows from both tables.

=== WHERE Clauses ===

If you want to select data that satisfies a condition, use a '''WHERE''' clause.

A simple '''WHERE''' clause can use equality, less than, greater than, and most other comparison operators. For example, the following query selects all last names of employees with an Employee ID of less than 10:

<source lang="mysql">
select
last_name
from employees
where
employee_id < 10;
</source>

If we wanted to select everyone from our employees table whose name started with a J, we could use a '''LIKE''' statement in our '''WHERE''' clause:

<source lang="mysql">
select
first_name,
last_name
from employees
where
first_name like 'J%';
</source>

The '''LIKE''' clause is SQL's way of letting you perform simple tests on strings. (SQL's built-in string comparison operations are not as powerful as regular expressions, but they'll get the job done in most everyday situations.)

=== Sorting Data ===

You can sort data using an '''ORDER BY''' clause like so:

<source lang="mysql">
select
first_name,
last_name
from employees
order by last_name asc;
</source>

If you wanted to select only the first few results from a query, use the '''LIMIT''' clause:

<source lang="mysql">
select
first_name,
last_name
from employees
order by last_name asc
limit 10;
</source>

It is important to note that the ORDER BY on a field is performed ''after'' aggregation functions. Currently, there is no way in MySQL to perform sorting ''before'' aggregation has occurred in a single query; we need to resort to nested queries. (For example, in the baseball example below, it's not easy to select "the player from each team with the highest batting average". It's even harder, if not impossible, to select "the ''three'' players from each team with the highest batting averages".)

=== Aggregation Clauses ===

When performing computations on an entire database, you will probably find yourself wanting to use ''aggregation functions''. Aggregation functions include:

* '''AVG(column_name)''' to compute the average value of column_name
* '''STD(column_name)''' to compute the standard deviation of column_name
* '''SUM(column_name)''' to compute the sum
* '''MIN(column_name)''' to find the minimum value
* '''MAX(column_name)''' to fund the maximum value
* '''COUNT(column_name)''' to count the total number of entries having a not null entry in column_name

For example, if you had a table containing batter averages, you could calculate the mean batter average using this query:

<source lang="mysql">
select
avg(batter_average) as avg_batter_average
from baseball_players;
</source>

Since WHERE clauses are performed ''before'' aggregation, you can also perform a query like

<source lang="mysql">
select
avg(batter_average) as avg_batter_average
from baseball_players
where
team_id=5;
</source>

If you want to aggregate all rows that share a certain value in an index, use a '''GROUP BY''' clause. For example, to select the average batter average for all teams, you could do

<source lang="mysql">
select
avg(batter_average) as avg_batter_average
from baseball_players
group by team_id;
</source>

You may find yourself wanting to apply a condition after the aggregation has occurred. You can do this using a '''HAVING''' clause. For example, the following query would select all baseball teams whose average batter average is more than 300:

<source lang="mysql">
select
avg(batter_average) as avg_batter_average
from baseball_players
group by team_id
having
avg_batter_average > 300;
</source>

It is perfectly valid to combine WHERE and HAVING clauses in a single query. For example, the following query first finds all players whose first name starts with "J", averages their batting averages together by team, and then selects those teams whose average batting average of the "J" players is more than 300; that is, it selects those teams "whose average batting average of players whose first name starts with J is more than 300":

<source lang="mysql">
select
avg(batter_average) as avg_batter_average
from baseball_players
where
first_name like "J%"
group by team_id
having
avg_batter_average > 300;
</source>

[[Category:Module 3]]

MySQL Schema and State

2023-01-10T06:05:33Z

Zach: /* Foreign Key */

MySQL is a powerful database that allows for complex logic when manipulating data. This guide is an introduction and a reference to common SQL queries for schema and state manipulation.

This guide assumes that you already have a working copy of MySQL. If you don't, read [[Introduction to MySQL]] first.

==Schema ==

The ''schema'' of your database refers to the way that your data is structured and to how bits of data relate to each other. Every database in your MySQL server has a separate schema, and separate databases should not typically interact with one another.

=== Tables ===

A database contains one or more ''tables''. Tables contain columns and rows, which you may sometimes call ''fields'' and ''entries''.

You can see all of the tables in the database using this query:

<source lang="mysql">
show tables;
</source>

When you create a table, you should provide information about all of that table's columns within the ''create table'' query. Here is the general syntax for the query:

<source lang="mysql">
create table <TABLENAME> (
<FIELDNAME> <DATATYPE> <OPTIONS>,
<FIELDNAME> <DATATYPE> <OPTIONS>,
<FIELDNAME> <DATATYPE> <OPTIONS>,
<FIELDNAME> <DATATYPE> <OPTIONS>
) <TABLE_OPTIONS>;
</source>

You can get details about an existing table using this query:

<source lang="mysql">
describe <TABLENAME>;
</source>

==== Data Types ====

Just like a programming language, MySQL has data types. There are dozens of data types; some of the most common ones are documented below.

===== String Data Types =====

There are four main types of string data types: character, text, blob, and enumerable.

* Character types are useful for short strings, like usernames and e-mail addresses.
* Text types differ from character types in that behind the scenes, they are not actually stored with the rest of the table row; they are useful for longer blocks of text, like forum posts.
* Blobs are for binary data, like files.
* Enumerables are for something that could have one of a set number of values (not more than 65535), like what State a user is from (Alabama, Alaska, and so on). Enums are also useful for storing boolean data, like '''ENUM('yes','no')'''. An enum is actually stored as an integer in the database, making queries more efficient.

The syntax for these data types is:

* '''VARCHAR(#)''', a string ''at most'' # characters long, where 0 ≤ # ≤ 65535
* '''CHAR(#)''', a string ''exactly'' # characters long, where 0 ≤ # ≤ 65535
* '''TINYTEXT''', a text block at most 255 characters long
* '''TEXT''', a text block at most 65535 characters long
* '''LONGTEXT''', a text block at most 4 GB in length
* '''BLOB''', a binary string at most 65 KB in length
* '''MEDIUMBLOB''', a binary string at most 16 MB in length
* '''LONGBLOB''', a binary string at most 4 GB in length
* '''ENUM('a', 'b', 'c')''', an enumerable with choices a, b, and c

===== Numeric Data Types =====

There are two types of numeric data types that you need to worry about: integers and decimals. The difference between the two should be pretty self-explanatory. The syntax for using these data types is:

* '''TINYINT''', an integer ''n'' where -128 ≤ n ≤ 127
* '''TINYINT UNSIGNED''', an integer ''n'' where 0 ≤ n ≤ 255
* '''SMALLINT''', an integer ''n'' where -32,768 ≤ n ≤ 32,767
* '''SMALLINT UNSIGNED''', an integer ''n'' where 0 ≤ n ≤ 65,535
* '''MEDIUMINT''', an integer ''n'' where -8,388,608 ≤ n ≤ 8,388,607
* '''MEDIUMINT UNSIGNED''', an integer ''n'' where 0 ≤ n ≤ 16,777,215
* '''INT''', an integer ''n'' where -2,147,483,648 ≤ n ≤ 2,147,483,647
* '''INT UNSIGNED''', an integer ''n'' where 0 ≤ n ≤ 4,294,967,295
* '''BIGINT''', an integer ''n'' where -9,223,372,036,854,775,808 ≤ n ≤ 9,223,372,036,854,775,807
* '''BIGINT UNSIGNED''', an integer ''n'' where 0 ≤ n ≤ 18,446,744,073,709,551,615
* '''DECIMAL(#a, #b)''', a decimal number at most ''#a'' digits long, ''#b'' of which are after the decimal point. For example, ''DECIMAL(5,2)'' can contain ''n'' where -999.99 ≤ n ≤ 999.99

When choosing the size of an integer, you should use the ''smallest integer for which you know you will never "run out of room"''. In most everyday circumstances, '''MEDIUMINT''' should be the biggest you'll need.

===== Temporal Data Types =====
{{XKCD
|name=2038
|id=607
}}

MySQL provides a class of data types that specialize in handling dates and times. The syntax for using date types is:

* '''DATE''', a date ''d'' where <code>1000-01-01</code> ≤ d ≤ <code>9999-12-31</code>
* '''DATETIME''', a date with time ''d'' where <code>1000-01-01 00:00:00</code> ≤ d ≤ <code>9999-12-31 23:59:59</code>
* '''TIMESTAMP''', a date with time ''d'' where <code>1970-01-01 00:00:01 UTC</code> ≤ d ≤ <code>2038-01-19 03:14:07 UTC</code> (note: <code>1970-01-01 00:00:01 UTC</code> is the Unix epoch)
* '''TIME''', a length of time ''t'' where <code>-838:59:59</code> ≤ t ≤ <code>838:59:59</code> (note: you cannot store time to a precision greater than 1 second)
* '''YEAR''', a year ''y'' where 1901 ≤ y ≤ 2155

Note that '''TIMESTAMP''' is a highly versatile data type that can automatically change whenever a row is updated. For more information, refer to [http://dev.mysql.com/doc/refman/5.6/en/timestamp-initialization.html the MySQL documentation].

==== Field Options ====

By default, any "cell" in a MySQL table can contain the ''NULL'' value. If a certain field in your database schema will never contain a null value, you should declare it as '''NOT NULL''' in order to make queries more efficient.

You can also specify a default value for a field. If you do, when you insert data into the table, it won't be necessary to specify a value for that column. To specify a default value for a field, use the syntax '''DEFAULT 'DEFAULT-VALUE' '''

==== Create Table Example ====

The following query would create a table called ''employees'' with six columns: employee ID, first name, last name, nickname, department, and the date that they joined the company. Nickname is the only optional column. The primary key for the table (see the next section) will be the ''id'' column. The department will be one of five choices, with the default being CSE. The storage engine for the table will be InnoDB, and the default charset for the table will be UTF8.

<source lang="mysql">
create table employees (
id mediumint unsigned not null auto_increment,
first_name varchar(30) not null,
last_name varchar(40) not null,
nickname varchar(20),
department enum('CSE','BME','EECE','ESE','MEMS') not null default 'CSE',
joined timestamp not null default current_timestamp,
primary key (id)
) engine = InnoDB default character set = utf8 collate = utf8_general_ci;
</source>

=== Keys ===

When MySQL goes to get data out of a database, it does so using an index, like the index of a textbook. In tech talk, we call indices ''keys''.

==== Primary Key ====

The '''primary key''' is what MySQL uses to distinguish one row from another. Every table must have exactly one primary key; no more, no less.

It is common practice to add a column to a table with data type '''mediumint unsigned not null auto_increment''' to serve as the primary key for a table. The '''auto_increment''' is a neat little feature of MySQL that basically says, "when a new row is inserted into this table, assign it the next-lowest available ID".

To specify a primary key for a table, use the syntax:

<source lang="mysql">
primary key (column1, column2, column3)
</source>

==== Unique Key ====

If you want to specify that all items in a certain column will be unique from each other (like usernames or e-mail addresses), you can use a '''unique key'''.

Because a table can have more than one unique key, you need to give your unique key a name. It is common practice to name the key after the columns it is indexing, like ''idx_unique_column1_column2''.

The syntax for specifying a unique key is:

<source lang="mysql">
unique key <INDEX_NAME> (<column1>, <column2>, <column3>)
</source>

==== Foreign Key ====

A '''foreign key''' is the most complicated type of MySQL index. A foreign key says that the values in one column or columns are associated with the values in a column or columns in a ''different table''.

For example, suppose you had two tables, one containing users and the other containing posts on a forum. There is a one-to-many relationship between users and posts. You could link a post to a user by having a column in your post table called ''user_id'', which would be a ''foreign key'' to the ''id'' column in your users table.

Foreign keys can be a source of great confusion in developing database schemas. If you get stuck, keep this tip in mind: '''Foreign Keys should always reference the columns used in the Primary Key of a different table.'''

The syntax for creating a foreign key is as follows:

<source lang="mysql">
foreign key (column1, column2, column3) references FOREIGN_TABLENAME (column1, column2, column3)
</source>

For example, in the user-and-post example above, the foreign key declaration would be:

<source lang="mysql">
foreign key (user_id) references user (id)
</source>

===== Further Reading about Foreign Keys =====

* [[wikipedia:Foreign Key|Wikipedia Entry on Foreign Keys]]
* [http://www.techrepublic.com/article/an-introduction-to-foreign-keys-and-referential-integrity-in-mysql/6035435 An introductory article on Tech Republic]
* [http://dev.mysql.com/doc/refman/5.1/en/innodb-foreign-key-constraints.html MySQL Documentation for Foreign Keys]

==== Syntax for Keys ====

You can define keys in your '''create table''' query, or you can define them later.

If you define them in your '''create table''' query, do so after the column definitions but still inside the parentheses:

<source lang="mysql">
create table TABLENAME (
column,
column,
....,
primary key (column1, column2, ...),
unique key INDEXNAME (column1, column2, ...),
foreign key (column1, column2, ...) references FOREIGN_TABLENAME (column1, column2, ...)
)
</source>

To define indices later, use an '''ALTER TABLE''' query:

<source lang="mysql">
alter table TABLENAME add unique key INDEXNAME (column1, column2, ...);
alter table TABLENAME add foreign key (column1, column2, ...) references FOREIGN_TABLENAME (column1, column2, ...);
</source>

== State ==

The ''state'' of your database refers to the data that it holds. Whenever you insert, update, or delete data, the state of your database changes.

=== Inserting Data from a Query ===

To insert a row (aka ''entry'') into a table, use a query of the form

<source lang="mysql">
insert into TABLENAME (column1, column2, column3) values (value1_1, value2_1, value3_1), (value1_2, value2_2, value3_2), ...
</source>

For example, to insert an employee into the table we created earlier, you could use the query

<source lang="mysql">
insert into employees (first_name, last_name, department) values ('John', 'Doe', 'BME'), ('Keith', 'Jones', 'ESE');
</source>

Because of the various default and auto_increment values we have specified in our database, the database would now contain the following data:

+----+------------+-----------+----------+------------+---------------------+
| id | first_name | last_name | nickname | department | joined |
+----+------------+-----------+----------+------------+---------------------+
| 1 | John | Doe | NULL | BME | 2012-08-22 22:02:24 |
| 2 | Keith | Jones | NULL | ESE | 2012-08-22 22:02:24 |
+----+------------+-----------+----------+------------+---------------------+

If you want to insert a DATE or DATETIME value into a database, use the YYYY-MM-DD HH:MM:SS format. The exception is with TIMESTAMP, in which you should use a 16-bit integer representing the number of seconds since January 1, 1970.

=== Inserting Data from a File ===

The '''load data infile''' command populates a table using data stored in a text file. The general form of the command is:

<source lang="mysql">
LOAD DATA INFILE 'dataFile.txt' INTO TABLE table_name;
</source>

where ''dataFile.txt'' is located relative to the server's data directory if a path name with leading components is given; otherwise, if no leading components are given, the server assumes the file is in the default database's database directory. If an absolute path is specified, the server uses it. For this command to work, you must issue the following command to grant the FILE privilege to your user:

<source lang="mysql">
GRANT FILE on *.* to 'user'@'host';
</source>

Alternatively, and quite usefully, you can load data from a local file (a file residing on the same machine as the client) using the LOCAL keyword. When loading data from the local machine you do not need to grant your user the FILE privilege, think about why this might be the case. (Hint: what is going on behind the scenes when you load the data in from a local machine when your database is hosted on a different machine?)

<source lang="mysql">
LOAD DATA LOCAL INFILE '/home/linus_torvalds/Documents/dataFile.txt' INTO TABLE table_name;
</source>

'''Note:''' When the local keyword is used, it is best to specify an absolute path.

'''Additional Note:''' If we SSH into our EC2 instance and then run the mysql command, the client is the EC2 instance, not the local computer that we SSH from. The LOCAL keyword in this case is allowing us to load files from the EC2 instance, not the local computer that we are using.

By default, the text files are assumed to have tab-delimited columns with column values specified in the same order as they are defined in the table (though you can specify additional/alternative delimiters using additional options).

=== Updating Data ===

To update a row in a table, use a query of the form

<source lang="mysql">
update TABLENAME set column1=value1, column2=value2, column3=value3 where CONDITION
</source>

For example, to set Keith Jones's nickname to "KJ", you would run the query

<source lang="mysql">
update employees set nickname='KJ' where id=2;
</source>

=== Deleting Data ===

To delete a row from a table, use a query of the form

<source lang="mysql">
delete from TABLENAME where CONDITION
</source>

For example, to delete John Doe from the table, you would run the query

<source lang="mysql">
delete from employees where id=1;
</source>

'''DO NOT FORGET THE WHERE CLAUSE!''' If you do, ''all'' rows will be deleted from your table!

== Selecting Data ==

The most simple query to select data from a table is:

<source lang="mysql">
select * from TABLENAME;
</source>

However, in practice, you will almost always want to perform a more advanced query.

=== Selecting Specific Columns ===

In order to save transmission time, you should select only those columns that you need in your application. The syntax for selecting only certain columns is

<source lang="mysql">
select column1, column2, column3 from TABLENAME;
</source>

=== Computation ===

MySQL can perform computation. For example, if you wanted to select the amount of time that a certain employee has been at the company, you could run the query

<source lang="mysql">
select TIMESTAMPDIFF( second, joined, NOW() ) as time_with_company from employees;
</source>

=== Joins ===

When you have one-to-one, one-to-many, or many-to-many relationships between tables, you will often want to select data from multiple tables in one query. To do this, use '''joins'''.

Joins are best described by example. Suppose you had the following tables in a MySQL database:

mysql> describe employees;
+------------+---------------------------------------+------+-----+-------------------+----------------+
| Field | Type | Null | Key | Default | Extra |
+------------+---------------------------------------+------+-----+-------------------+----------------+
| id | mediumint(8) unsigned | NO | PRI | NULL | auto_increment |
| first_name | varchar(30) | NO | | NULL | |
| last_name | varchar(40) | NO | | NULL | |
| nickname | varchar(20) | YES | | NULL | |
| department | enum('CSE','BME','EECE','ESE','MEMS') | NO | | CSE | |
| joined | timestamp | NO | | CURRENT_TIMESTAMP | |
+------------+---------------------------------------+------+-----+-------------------+----------------+
6 rows in set (0.00 sec)

mysql> describe projects;
+--------------+-----------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+--------------+-----------------------+------+-----+---------+----------------+
| id | mediumint(8) unsigned | NO | PRI | NULL | auto_increment |
| employee_id | mediumint(8) unsigned | NO | MUL | NULL | |
| project_name | tinytext | NO | | NULL | |
+--------------+-----------------------+------+-----+---------+----------------+
3 rows in set (0.00 sec)

Further, suppose the tables had the following data:

mysql> select * from employees;
+----+------------+-----------+----------+------------+---------------------+
| id | first_name | last_name | nickname | department | joined |
+----+------------+-----------+----------+------------+---------------------+
| 1 | John | Doe | NULL | BME | 2012-08-22 22:02:24 |
| 2 | Keith | Jones | KK | ESE | 2012-08-22 22:02:24 |
+----+------------+-----------+----------+------------+---------------------+
2 rows in set (0.00 sec)

mysql> select * from projects;
+----+-------------+-----------------------------------+
| id | employee_id | project_name |
+----+-------------+-----------------------------------+
| 1 | 1 | Interview the Chancellor |
| 2 | 1 | Give Presentation in Mumbai |
| 3 | 2 | Publish Research Paper in Journal |
+----+-------------+-----------------------------------+
3 rows in set (0.00 sec)

If you ran the following query:

<source lang="mysql">
select
project_name,
employees.first_name,
employees.last_name
from projects
join employees on (projects.employee_id=employees.id);
</source>

You would get the result:

mysql> select project_name, employees.first_name, employees.last_name from projects
-> join employees on (projects.employee_id=employees.id);
+-----------------------------------+------------+-----------+
| project_name | first_name | last_name |
+-----------------------------------+------------+-----------+
| Interview the Chancellor | John | Doe |
| Give Presentation in Mumbai | John | Doe |
| Publish Research Paper in Journal | Keith | Jones |
+-----------------------------------+------------+-----------+
3 rows in set (0.00 sec)

Isn't that cool?

==== Inner Joins, Left Joins, Right Joins, and Full Joins ====

When you perform a '''JOIN''' like we did above, you are implicitly performing an '''INNER JOIN'''. However, there may be times when you want to select data in a query even if it doesn't have a complement in the joined table. For example, suppose Keith Jones cancelled his project. The projects table now contains the following data:

mysql> select * from projects;
+----+-------------+-----------------------------+
| id | employee_id | project_name |
+----+-------------+-----------------------------+
| 1 | 1 | Interview the Chancellor |
| 2 | 1 | Give Presentation in Mumbai |
+----+-------------+-----------------------------+
2 rows in set (0.00 sec)

Now, we could run the following query:

<source lang="mysql">
select
count(projects.id) as project_num,
employees.first_name,
employees.last_name
from employees
join projects on (employees.id=projects.employee_id)
group by employees.id;
</source>

We get the result:

mysql> select count(projects.id) as project_num, employees.first_name, employees.last_name from employees
-> join projects on (employees.id=projects.employee_id) group by employees.id;
+-------------+------------+-----------+
| project_num | first_name | last_name |
+-------------+------------+-----------+
| 2 | John | Doe |
+-------------+------------+-----------+
1 row in set (0.00 sec)

But where's Keith Jones? Since Keith Jones didn't have any projects, he was omitted from the result. This is probably not what we intended to happen. Instead, you should perform a '''LEFT JOIN''', like so:

<source lang="mysql">
select
count(projects.id) as project_num,
employees.first_name,
employees.last_name
from employees
left join projects on (employees.id=projects.employee_id)
group by employees.id;
</source>

The result will now be:

mysql> select count(projects.id) as project_num, employees.first_name, employees.last_name from employees
-> left join projects on (employees.id=projects.employee_id) group by employees.id;
+-------------+------------+-----------+
| project_num | first_name | last_name |
+-------------+------------+-----------+
| 2 | John | Doe |
| 0 | Keith | Jones |
+-------------+------------+-----------+
2 rows in set (0.00 sec)

(For more information on COUNT and GROUP BY, see [[#Aggregation Clauses|a later section]].)

You can see that a LEFT JOIN preserves all rows from the "left" table in the join. Likewise, a RIGHT JOIN preserves all rows from the "right" table. A FULL JOIN preserves all rows from both tables.

=== WHERE Clauses ===

If you want to select data that satisfies a condition, use a '''WHERE''' clause.

A simple '''WHERE''' clause can use equality, less than, greater than, and most other comparison operators. For example, the following query selects all last names of employees with an Employee ID of less than 10:

<source lang="mysql">
select
last_name
from employees
where
employee_id < 10;
</source>

If we wanted to select everyone from our employees table whose name started with a J, we could use a '''LIKE''' statement in our '''WHERE''' clause:

<source lang="mysql">
select
first_name,
last_name
from employees
where
first_name like 'J%';
</source>

The '''LIKE''' clause is SQL's way of letting you perform simple tests on strings. (SQL's built-in string comparison operations are not as powerful as regular expressions, but they'll get the job done in most everyday situations.)

=== Sorting Data ===

You can sort data using an '''ORDER BY''' clause like so:

<source lang="mysql">
select
first_name,
last_name
from employees
order by last_name asc;
</source>

If you wanted to select only the first few results from a query, use the '''LIMIT''' clause:

<source lang="mysql">
select
first_name,
last_name
from employees
order by last_name asc
limit 10;
</source>

It is important to note that the ORDER BY on a field is performed ''after'' aggregation functions. Currently, there is no way in MySQL to perform sorting ''before'' aggregation has occurred in a single query; we need to resort to nested queries. (For example, in the baseball example below, it's not easy to select "the player from each team with the highest batting average". It's even harder, if not impossible, to select "the ''three'' players from each team with the highest batting averages".)

=== Aggregation Clauses ===

When performing computations on an entire database, you will probably find yourself wanting to use ''aggregation functions''. Aggregation functions include:

* '''AVG(column_name)''' to compute the average value of column_name
* '''STD(column_name)''' to compute the standard deviation of column_name
* '''SUM(column_name)''' to compute the sum
* '''MIN(column_name)''' to find the minimum value
* '''MAX(column_name)''' to fund the maximum value
* '''COUNT(column_name)''' to count the total number of entries having a not null entry in column_name

For example, if you had a table containing batter averages, you could calculate the mean batter average using this query:

<source lang="mysql">
select
avg(batter_average) as avg_batter_average
from baseball_players;
</source>

Since WHERE clauses are performed ''before'' aggregation, you can also perform a query like

<source lang="mysql">
select
avg(batter_average) as avg_batter_average
from baseball_players
where
team_id=5;
</source>

If you want to aggregate all rows that share a certain value in an index, use a '''GROUP BY''' clause. For example, to select the average batter average for all teams, you could do

<source lang="mysql">
select
avg(batter_average) as avg_batter_average
from baseball_players
group by team_id;
</source>

You may find yourself wanting to apply a condition after the aggregation has occurred. You can do this using a '''HAVING''' clause. For example, the following query would select all baseball teams whose average batter average is more than 300:

<source lang="mysql">
select
avg(batter_average) as avg_batter_average
from baseball_players
group by team_id
having
avg_batter_average > 300;
</source>

It is perfectly valid to combine WHERE and HAVING clauses in a single query. For example, the following query first finds all players whose first name starts with "J", averages their batting averages together by team, and then selects those teams whose average batting average of the "J" players is more than 300; that is, it selects those teams "whose average batting average of players whose first name starts with J is more than 300":

<source lang="mysql">
select
avg(batter_average) as avg_batter_average
from baseball_players
where
first_name like "J%"
group by team_id
having
avg_batter_average > 300;
</source>

[[Category:Module 3]]

MySQL Schema and State

2023-01-10T06:04:42Z

Zach: /* Syntax for Keys */

MySQL is a powerful database that allows for complex logic when manipulating data. This guide is an introduction and a reference to common SQL queries for schema and state manipulation.

This guide assumes that you already have a working copy of MySQL. If you don't, read [[Introduction to MySQL]] first.

==Schema ==

The ''schema'' of your database refers to the way that your data is structured and to how bits of data relate to each other. Every database in your MySQL server has a separate schema, and separate databases should not typically interact with one another.

=== Tables ===

A database contains one or more ''tables''. Tables contain columns and rows, which you may sometimes call ''fields'' and ''entries''.

You can see all of the tables in the database using this query:

<source lang="mysql">
show tables;
</source>

When you create a table, you should provide information about all of that table's columns within the ''create table'' query. Here is the general syntax for the query:

<source lang="mysql">
create table <TABLENAME> (
<FIELDNAME> <DATATYPE> <OPTIONS>,
<FIELDNAME> <DATATYPE> <OPTIONS>,
<FIELDNAME> <DATATYPE> <OPTIONS>,
<FIELDNAME> <DATATYPE> <OPTIONS>
) <TABLE_OPTIONS>;
</source>

You can get details about an existing table using this query:

<source lang="mysql">
describe <TABLENAME>;
</source>

==== Data Types ====

Just like a programming language, MySQL has data types. There are dozens of data types; some of the most common ones are documented below.

===== String Data Types =====

There are four main types of string data types: character, text, blob, and enumerable.

* Character types are useful for short strings, like usernames and e-mail addresses.
* Text types differ from character types in that behind the scenes, they are not actually stored with the rest of the table row; they are useful for longer blocks of text, like forum posts.
* Blobs are for binary data, like files.
* Enumerables are for something that could have one of a set number of values (not more than 65535), like what State a user is from (Alabama, Alaska, and so on). Enums are also useful for storing boolean data, like '''ENUM('yes','no')'''. An enum is actually stored as an integer in the database, making queries more efficient.

The syntax for these data types is:

* '''VARCHAR(#)''', a string ''at most'' # characters long, where 0 ≤ # ≤ 65535
* '''CHAR(#)''', a string ''exactly'' # characters long, where 0 ≤ # ≤ 65535
* '''TINYTEXT''', a text block at most 255 characters long
* '''TEXT''', a text block at most 65535 characters long
* '''LONGTEXT''', a text block at most 4 GB in length
* '''BLOB''', a binary string at most 65 KB in length
* '''MEDIUMBLOB''', a binary string at most 16 MB in length
* '''LONGBLOB''', a binary string at most 4 GB in length
* '''ENUM('a', 'b', 'c')''', an enumerable with choices a, b, and c

===== Numeric Data Types =====

There are two types of numeric data types that you need to worry about: integers and decimals. The difference between the two should be pretty self-explanatory. The syntax for using these data types is:

* '''TINYINT''', an integer ''n'' where -128 ≤ n ≤ 127
* '''TINYINT UNSIGNED''', an integer ''n'' where 0 ≤ n ≤ 255
* '''SMALLINT''', an integer ''n'' where -32,768 ≤ n ≤ 32,767
* '''SMALLINT UNSIGNED''', an integer ''n'' where 0 ≤ n ≤ 65,535
* '''MEDIUMINT''', an integer ''n'' where -8,388,608 ≤ n ≤ 8,388,607
* '''MEDIUMINT UNSIGNED''', an integer ''n'' where 0 ≤ n ≤ 16,777,215
* '''INT''', an integer ''n'' where -2,147,483,648 ≤ n ≤ 2,147,483,647
* '''INT UNSIGNED''', an integer ''n'' where 0 ≤ n ≤ 4,294,967,295
* '''BIGINT''', an integer ''n'' where -9,223,372,036,854,775,808 ≤ n ≤ 9,223,372,036,854,775,807
* '''BIGINT UNSIGNED''', an integer ''n'' where 0 ≤ n ≤ 18,446,744,073,709,551,615
* '''DECIMAL(#a, #b)''', a decimal number at most ''#a'' digits long, ''#b'' of which are after the decimal point. For example, ''DECIMAL(5,2)'' can contain ''n'' where -999.99 ≤ n ≤ 999.99

When choosing the size of an integer, you should use the ''smallest integer for which you know you will never "run out of room"''. In most everyday circumstances, '''MEDIUMINT''' should be the biggest you'll need.

===== Temporal Data Types =====
{{XKCD
|name=2038
|id=607
}}

MySQL provides a class of data types that specialize in handling dates and times. The syntax for using date types is:

* '''DATE''', a date ''d'' where <code>1000-01-01</code> ≤ d ≤ <code>9999-12-31</code>
* '''DATETIME''', a date with time ''d'' where <code>1000-01-01 00:00:00</code> ≤ d ≤ <code>9999-12-31 23:59:59</code>
* '''TIMESTAMP''', a date with time ''d'' where <code>1970-01-01 00:00:01 UTC</code> ≤ d ≤ <code>2038-01-19 03:14:07 UTC</code> (note: <code>1970-01-01 00:00:01 UTC</code> is the Unix epoch)
* '''TIME''', a length of time ''t'' where <code>-838:59:59</code> ≤ t ≤ <code>838:59:59</code> (note: you cannot store time to a precision greater than 1 second)
* '''YEAR''', a year ''y'' where 1901 ≤ y ≤ 2155

Note that '''TIMESTAMP''' is a highly versatile data type that can automatically change whenever a row is updated. For more information, refer to [http://dev.mysql.com/doc/refman/5.6/en/timestamp-initialization.html the MySQL documentation].

==== Field Options ====

By default, any "cell" in a MySQL table can contain the ''NULL'' value. If a certain field in your database schema will never contain a null value, you should declare it as '''NOT NULL''' in order to make queries more efficient.

You can also specify a default value for a field. If you do, when you insert data into the table, it won't be necessary to specify a value for that column. To specify a default value for a field, use the syntax '''DEFAULT 'DEFAULT-VALUE' '''

==== Create Table Example ====

The following query would create a table called ''employees'' with six columns: employee ID, first name, last name, nickname, department, and the date that they joined the company. Nickname is the only optional column. The primary key for the table (see the next section) will be the ''id'' column. The department will be one of five choices, with the default being CSE. The storage engine for the table will be InnoDB, and the default charset for the table will be UTF8.

<source lang="mysql">
create table employees (
id mediumint unsigned not null auto_increment,
first_name varchar(30) not null,
last_name varchar(40) not null,
nickname varchar(20),
department enum('CSE','BME','EECE','ESE','MEMS') not null default 'CSE',
joined timestamp not null default current_timestamp,
primary key (id)
) engine = InnoDB default character set = utf8 collate = utf8_general_ci;
</source>

=== Keys ===

When MySQL goes to get data out of a database, it does so using an index, like the index of a textbook. In tech talk, we call indices ''keys''.

==== Primary Key ====

The '''primary key''' is what MySQL uses to distinguish one row from another. Every table must have exactly one primary key; no more, no less.

It is common practice to add a column to a table with data type '''mediumint unsigned not null auto_increment''' to serve as the primary key for a table. The '''auto_increment''' is a neat little feature of MySQL that basically says, "when a new row is inserted into this table, assign it the next-lowest available ID".

To specify a primary key for a table, use the syntax:

<source lang="mysql">
primary key (column1, column2, column3)
</source>

==== Unique Key ====

If you want to specify that all items in a certain column will be unique from each other (like usernames or e-mail addresses), you can use a '''unique key'''.

Because a table can have more than one unique key, you need to give your unique key a name. It is common practice to name the key after the columns it is indexing, like ''idx_unique_column1_column2''.

The syntax for specifying a unique key is:

<source lang="mysql">
unique key <INDEX_NAME> (<column1>, <column2>, <column3>)
</source>

==== Foreign Key ====

A '''foreign key''' is the most complicated type of MySQL index. A foreign key says that the values in one column or columns are associated with the values in a column or columns in a ''different table''.

For example, suppose you had two tables, one containing users and the other containing posts on a forum. There is a one-to-many relationship between users and posts. You could link a post to a user by having a column in your post table called ''user_id'', which would be a ''foreign key'' to the ''id'' column in your users table.

Foreign keys can be a source of great confusion in developing database schemas. If you get stuck, keep this tip in mind: '''Foreign Keys should always reference the columns used in the Primary Key of a different table.'''

The syntax for creating a foreign key is as follows:

<source lang="mysql">
foreign key (<column1>, <column2>, <column3>) references <FOREIGN_TABLENAME> (<column1>, <column2>, <column3>)
</source>

For example, in the user-and-post example above, the foreign key declaration would be:

<source lang="mysql">
foreign key (user_id) references user (id)
</source>

===== Further Reading about Foreign Keys =====

* [[wikipedia:Foreign Key|Wikipedia Entry on Foreign Keys]]
* [http://www.techrepublic.com/article/an-introduction-to-foreign-keys-and-referential-integrity-in-mysql/6035435 An introductory article on Tech Republic]
* [http://dev.mysql.com/doc/refman/5.1/en/innodb-foreign-key-constraints.html MySQL Documentation for Foreign Keys]

==== Syntax for Keys ====

You can define keys in your '''create table''' query, or you can define them later.

If you define them in your '''create table''' query, do so after the column definitions but still inside the parentheses:

<source lang="mysql">
create table TABLENAME (
column,
column,
....,
primary key (column1, column2, ...),
unique key INDEXNAME (column1, column2, ...),
foreign key (column1, column2, ...) references FOREIGN_TABLENAME (column1, column2, ...)
)
</source>

To define indices later, use an '''ALTER TABLE''' query:

<source lang="mysql">
alter table TABLENAME add unique key INDEXNAME (column1, column2, ...);
alter table TABLENAME add foreign key (column1, column2, ...) references FOREIGN_TABLENAME (column1, column2, ...);
</source>

== State ==

The ''state'' of your database refers to the data that it holds. Whenever you insert, update, or delete data, the state of your database changes.

=== Inserting Data from a Query ===

To insert a row (aka ''entry'') into a table, use a query of the form

<source lang="mysql">
insert into TABLENAME (column1, column2, column3) values (value1_1, value2_1, value3_1), (value1_2, value2_2, value3_2), ...
</source>

For example, to insert an employee into the table we created earlier, you could use the query

<source lang="mysql">
insert into employees (first_name, last_name, department) values ('John', 'Doe', 'BME'), ('Keith', 'Jones', 'ESE');
</source>

Because of the various default and auto_increment values we have specified in our database, the database would now contain the following data:

+----+------------+-----------+----------+------------+---------------------+
| id | first_name | last_name | nickname | department | joined |
+----+------------+-----------+----------+------------+---------------------+
| 1 | John | Doe | NULL | BME | 2012-08-22 22:02:24 |
| 2 | Keith | Jones | NULL | ESE | 2012-08-22 22:02:24 |
+----+------------+-----------+----------+------------+---------------------+

If you want to insert a DATE or DATETIME value into a database, use the YYYY-MM-DD HH:MM:SS format. The exception is with TIMESTAMP, in which you should use a 16-bit integer representing the number of seconds since January 1, 1970.

=== Inserting Data from a File ===

The '''load data infile''' command populates a table using data stored in a text file. The general form of the command is:

<source lang="mysql">
LOAD DATA INFILE 'dataFile.txt' INTO TABLE table_name;
</source>

where ''dataFile.txt'' is located relative to the server's data directory if a path name with leading components is given; otherwise, if no leading components are given, the server assumes the file is in the default database's database directory. If an absolute path is specified, the server uses it. For this command to work, you must issue the following command to grant the FILE privilege to your user:

<source lang="mysql">
GRANT FILE on *.* to 'user'@'host';
</source>

Alternatively, and quite usefully, you can load data from a local file (a file residing on the same machine as the client) using the LOCAL keyword. When loading data from the local machine you do not need to grant your user the FILE privilege, think about why this might be the case. (Hint: what is going on behind the scenes when you load the data in from a local machine when your database is hosted on a different machine?)

<source lang="mysql">
LOAD DATA LOCAL INFILE '/home/linus_torvalds/Documents/dataFile.txt' INTO TABLE table_name;
</source>

'''Note:''' When the local keyword is used, it is best to specify an absolute path.

'''Additional Note:''' If we SSH into our EC2 instance and then run the mysql command, the client is the EC2 instance, not the local computer that we SSH from. The LOCAL keyword in this case is allowing us to load files from the EC2 instance, not the local computer that we are using.

By default, the text files are assumed to have tab-delimited columns with column values specified in the same order as they are defined in the table (though you can specify additional/alternative delimiters using additional options).

=== Updating Data ===

To update a row in a table, use a query of the form

<source lang="mysql">
update TABLENAME set column1=value1, column2=value2, column3=value3 where CONDITION
</source>

For example, to set Keith Jones's nickname to "KJ", you would run the query

<source lang="mysql">
update employees set nickname='KJ' where id=2;
</source>

=== Deleting Data ===

To delete a row from a table, use a query of the form

<source lang="mysql">
delete from TABLENAME where CONDITION
</source>

For example, to delete John Doe from the table, you would run the query

<source lang="mysql">
delete from employees where id=1;
</source>

'''DO NOT FORGET THE WHERE CLAUSE!''' If you do, ''all'' rows will be deleted from your table!

== Selecting Data ==

The most simple query to select data from a table is:

<source lang="mysql">
select * from TABLENAME;
</source>

However, in practice, you will almost always want to perform a more advanced query.

=== Selecting Specific Columns ===

In order to save transmission time, you should select only those columns that you need in your application. The syntax for selecting only certain columns is

<source lang="mysql">
select column1, column2, column3 from TABLENAME;
</source>

=== Computation ===

MySQL can perform computation. For example, if you wanted to select the amount of time that a certain employee has been at the company, you could run the query

<source lang="mysql">
select TIMESTAMPDIFF( second, joined, NOW() ) as time_with_company from employees;
</source>

=== Joins ===

When you have one-to-one, one-to-many, or many-to-many relationships between tables, you will often want to select data from multiple tables in one query. To do this, use '''joins'''.

Joins are best described by example. Suppose you had the following tables in a MySQL database:

mysql> describe employees;
+------------+---------------------------------------+------+-----+-------------------+----------------+
| Field | Type | Null | Key | Default | Extra |
+------------+---------------------------------------+------+-----+-------------------+----------------+
| id | mediumint(8) unsigned | NO | PRI | NULL | auto_increment |
| first_name | varchar(30) | NO | | NULL | |
| last_name | varchar(40) | NO | | NULL | |
| nickname | varchar(20) | YES | | NULL | |
| department | enum('CSE','BME','EECE','ESE','MEMS') | NO | | CSE | |
| joined | timestamp | NO | | CURRENT_TIMESTAMP | |
+------------+---------------------------------------+------+-----+-------------------+----------------+
6 rows in set (0.00 sec)

mysql> describe projects;
+--------------+-----------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+--------------+-----------------------+------+-----+---------+----------------+
| id | mediumint(8) unsigned | NO | PRI | NULL | auto_increment |
| employee_id | mediumint(8) unsigned | NO | MUL | NULL | |
| project_name | tinytext | NO | | NULL | |
+--------------+-----------------------+------+-----+---------+----------------+
3 rows in set (0.00 sec)

Further, suppose the tables had the following data:

mysql> select * from employees;
+----+------------+-----------+----------+------------+---------------------+
| id | first_name | last_name | nickname | department | joined |
+----+------------+-----------+----------+------------+---------------------+
| 1 | John | Doe | NULL | BME | 2012-08-22 22:02:24 |
| 2 | Keith | Jones | KK | ESE | 2012-08-22 22:02:24 |
+----+------------+-----------+----------+------------+---------------------+
2 rows in set (0.00 sec)

mysql> select * from projects;
+----+-------------+-----------------------------------+
| id | employee_id | project_name |
+----+-------------+-----------------------------------+
| 1 | 1 | Interview the Chancellor |
| 2 | 1 | Give Presentation in Mumbai |
| 3 | 2 | Publish Research Paper in Journal |
+----+-------------+-----------------------------------+
3 rows in set (0.00 sec)

If you ran the following query:

<source lang="mysql">
select
project_name,
employees.first_name,
employees.last_name
from projects
join employees on (projects.employee_id=employees.id);
</source>

You would get the result:

mysql> select project_name, employees.first_name, employees.last_name from projects
-> join employees on (projects.employee_id=employees.id);
+-----------------------------------+------------+-----------+
| project_name | first_name | last_name |
+-----------------------------------+------------+-----------+
| Interview the Chancellor | John | Doe |
| Give Presentation in Mumbai | John | Doe |
| Publish Research Paper in Journal | Keith | Jones |
+-----------------------------------+------------+-----------+
3 rows in set (0.00 sec)

Isn't that cool?

==== Inner Joins, Left Joins, Right Joins, and Full Joins ====

When you perform a '''JOIN''' like we did above, you are implicitly performing an '''INNER JOIN'''. However, there may be times when you want to select data in a query even if it doesn't have a complement in the joined table. For example, suppose Keith Jones cancelled his project. The projects table now contains the following data:

mysql> select * from projects;
+----+-------------+-----------------------------+
| id | employee_id | project_name |
+----+-------------+-----------------------------+
| 1 | 1 | Interview the Chancellor |
| 2 | 1 | Give Presentation in Mumbai |
+----+-------------+-----------------------------+
2 rows in set (0.00 sec)

Now, we could run the following query:

<source lang="mysql">
select
count(projects.id) as project_num,
employees.first_name,
employees.last_name
from employees
join projects on (employees.id=projects.employee_id)
group by employees.id;
</source>

We get the result:

mysql> select count(projects.id) as project_num, employees.first_name, employees.last_name from employees
-> join projects on (employees.id=projects.employee_id) group by employees.id;
+-------------+------------+-----------+
| project_num | first_name | last_name |
+-------------+------------+-----------+
| 2 | John | Doe |
+-------------+------------+-----------+
1 row in set (0.00 sec)

But where's Keith Jones? Since Keith Jones didn't have any projects, he was omitted from the result. This is probably not what we intended to happen. Instead, you should perform a '''LEFT JOIN''', like so:

<source lang="mysql">
select
count(projects.id) as project_num,
employees.first_name,
employees.last_name
from employees
left join projects on (employees.id=projects.employee_id)
group by employees.id;
</source>

The result will now be:

mysql> select count(projects.id) as project_num, employees.first_name, employees.last_name from employees
-> left join projects on (employees.id=projects.employee_id) group by employees.id;
+-------------+------------+-----------+
| project_num | first_name | last_name |
+-------------+------------+-----------+
| 2 | John | Doe |
| 0 | Keith | Jones |
+-------------+------------+-----------+
2 rows in set (0.00 sec)

(For more information on COUNT and GROUP BY, see [[#Aggregation Clauses|a later section]].)

You can see that a LEFT JOIN preserves all rows from the "left" table in the join. Likewise, a RIGHT JOIN preserves all rows from the "right" table. A FULL JOIN preserves all rows from both tables.

=== WHERE Clauses ===

If you want to select data that satisfies a condition, use a '''WHERE''' clause.

A simple '''WHERE''' clause can use equality, less than, greater than, and most other comparison operators. For example, the following query selects all last names of employees with an Employee ID of less than 10:

<source lang="mysql">
select
last_name
from employees
where
employee_id < 10;
</source>

If we wanted to select everyone from our employees table whose name started with a J, we could use a '''LIKE''' statement in our '''WHERE''' clause:

<source lang="mysql">
select
first_name,
last_name
from employees
where
first_name like 'J%';
</source>

The '''LIKE''' clause is SQL's way of letting you perform simple tests on strings. (SQL's built-in string comparison operations are not as powerful as regular expressions, but they'll get the job done in most everyday situations.)

=== Sorting Data ===

You can sort data using an '''ORDER BY''' clause like so:

<source lang="mysql">
select
first_name,
last_name
from employees
order by last_name asc;
</source>

If you wanted to select only the first few results from a query, use the '''LIMIT''' clause:

<source lang="mysql">
select
first_name,
last_name
from employees
order by last_name asc
limit 10;
</source>

It is important to note that the ORDER BY on a field is performed ''after'' aggregation functions. Currently, there is no way in MySQL to perform sorting ''before'' aggregation has occurred in a single query; we need to resort to nested queries. (For example, in the baseball example below, it's not easy to select "the player from each team with the highest batting average". It's even harder, if not impossible, to select "the ''three'' players from each team with the highest batting averages".)

=== Aggregation Clauses ===

When performing computations on an entire database, you will probably find yourself wanting to use ''aggregation functions''. Aggregation functions include:

* '''AVG(column_name)''' to compute the average value of column_name
* '''STD(column_name)''' to compute the standard deviation of column_name
* '''SUM(column_name)''' to compute the sum
* '''MIN(column_name)''' to find the minimum value
* '''MAX(column_name)''' to fund the maximum value
* '''COUNT(column_name)''' to count the total number of entries having a not null entry in column_name

For example, if you had a table containing batter averages, you could calculate the mean batter average using this query:

<source lang="mysql">
select
avg(batter_average) as avg_batter_average
from baseball_players;
</source>

Since WHERE clauses are performed ''before'' aggregation, you can also perform a query like

<source lang="mysql">
select
avg(batter_average) as avg_batter_average
from baseball_players
where
team_id=5;
</source>

If you want to aggregate all rows that share a certain value in an index, use a '''GROUP BY''' clause. For example, to select the average batter average for all teams, you could do

<source lang="mysql">
select
avg(batter_average) as avg_batter_average
from baseball_players
group by team_id;
</source>

You may find yourself wanting to apply a condition after the aggregation has occurred. You can do this using a '''HAVING''' clause. For example, the following query would select all baseball teams whose average batter average is more than 300:

<source lang="mysql">
select
avg(batter_average) as avg_batter_average
from baseball_players
group by team_id
having
avg_batter_average > 300;
</source>

It is perfectly valid to combine WHERE and HAVING clauses in a single query. For example, the following query first finds all players whose first name starts with "J", averages their batting averages together by team, and then selects those teams whose average batting average of the "J" players is more than 300; that is, it selects those teams "whose average batting average of players whose first name starts with J is more than 300":

<source lang="mysql">
select
avg(batter_average) as avg_batter_average
from baseball_players
where
first_name like "J%"
group by team_id
having
avg_batter_average > 300;
</source>

[[Category:Module 3]]

MySQL Schema and State

2023-01-10T06:02:51Z

Zach: /* Syntax for Keys */

MySQL is a powerful database that allows for complex logic when manipulating data. This guide is an introduction and a reference to common SQL queries for schema and state manipulation.

This guide assumes that you already have a working copy of MySQL. If you don't, read [[Introduction to MySQL]] first.

==Schema ==

The ''schema'' of your database refers to the way that your data is structured and to how bits of data relate to each other. Every database in your MySQL server has a separate schema, and separate databases should not typically interact with one another.

=== Tables ===

A database contains one or more ''tables''. Tables contain columns and rows, which you may sometimes call ''fields'' and ''entries''.

You can see all of the tables in the database using this query:

<source lang="mysql">
show tables;
</source>

When you create a table, you should provide information about all of that table's columns within the ''create table'' query. Here is the general syntax for the query:

<source lang="mysql">
create table <TABLENAME> (
<FIELDNAME> <DATATYPE> <OPTIONS>,
<FIELDNAME> <DATATYPE> <OPTIONS>,
<FIELDNAME> <DATATYPE> <OPTIONS>,
<FIELDNAME> <DATATYPE> <OPTIONS>
) <TABLE_OPTIONS>;
</source>

You can get details about an existing table using this query:

<source lang="mysql">
describe <TABLENAME>;
</source>

==== Data Types ====

Just like a programming language, MySQL has data types. There are dozens of data types; some of the most common ones are documented below.

===== String Data Types =====

There are four main types of string data types: character, text, blob, and enumerable.

* Character types are useful for short strings, like usernames and e-mail addresses.
* Text types differ from character types in that behind the scenes, they are not actually stored with the rest of the table row; they are useful for longer blocks of text, like forum posts.
* Blobs are for binary data, like files.
* Enumerables are for something that could have one of a set number of values (not more than 65535), like what State a user is from (Alabama, Alaska, and so on). Enums are also useful for storing boolean data, like '''ENUM('yes','no')'''. An enum is actually stored as an integer in the database, making queries more efficient.

The syntax for these data types is:

* '''VARCHAR(#)''', a string ''at most'' # characters long, where 0 ≤ # ≤ 65535
* '''CHAR(#)''', a string ''exactly'' # characters long, where 0 ≤ # ≤ 65535
* '''TINYTEXT''', a text block at most 255 characters long
* '''TEXT''', a text block at most 65535 characters long
* '''LONGTEXT''', a text block at most 4 GB in length
* '''BLOB''', a binary string at most 65 KB in length
* '''MEDIUMBLOB''', a binary string at most 16 MB in length
* '''LONGBLOB''', a binary string at most 4 GB in length
* '''ENUM('a', 'b', 'c')''', an enumerable with choices a, b, and c

===== Numeric Data Types =====

There are two types of numeric data types that you need to worry about: integers and decimals. The difference between the two should be pretty self-explanatory. The syntax for using these data types is:

* '''TINYINT''', an integer ''n'' where -128 ≤ n ≤ 127
* '''TINYINT UNSIGNED''', an integer ''n'' where 0 ≤ n ≤ 255
* '''SMALLINT''', an integer ''n'' where -32,768 ≤ n ≤ 32,767
* '''SMALLINT UNSIGNED''', an integer ''n'' where 0 ≤ n ≤ 65,535
* '''MEDIUMINT''', an integer ''n'' where -8,388,608 ≤ n ≤ 8,388,607
* '''MEDIUMINT UNSIGNED''', an integer ''n'' where 0 ≤ n ≤ 16,777,215
* '''INT''', an integer ''n'' where -2,147,483,648 ≤ n ≤ 2,147,483,647
* '''INT UNSIGNED''', an integer ''n'' where 0 ≤ n ≤ 4,294,967,295
* '''BIGINT''', an integer ''n'' where -9,223,372,036,854,775,808 ≤ n ≤ 9,223,372,036,854,775,807
* '''BIGINT UNSIGNED''', an integer ''n'' where 0 ≤ n ≤ 18,446,744,073,709,551,615
* '''DECIMAL(#a, #b)''', a decimal number at most ''#a'' digits long, ''#b'' of which are after the decimal point. For example, ''DECIMAL(5,2)'' can contain ''n'' where -999.99 ≤ n ≤ 999.99

When choosing the size of an integer, you should use the ''smallest integer for which you know you will never "run out of room"''. In most everyday circumstances, '''MEDIUMINT''' should be the biggest you'll need.

===== Temporal Data Types =====
{{XKCD
|name=2038
|id=607
}}

MySQL provides a class of data types that specialize in handling dates and times. The syntax for using date types is:

* '''DATE''', a date ''d'' where <code>1000-01-01</code> ≤ d ≤ <code>9999-12-31</code>
* '''DATETIME''', a date with time ''d'' where <code>1000-01-01 00:00:00</code> ≤ d ≤ <code>9999-12-31 23:59:59</code>
* '''TIMESTAMP''', a date with time ''d'' where <code>1970-01-01 00:00:01 UTC</code> ≤ d ≤ <code>2038-01-19 03:14:07 UTC</code> (note: <code>1970-01-01 00:00:01 UTC</code> is the Unix epoch)
* '''TIME''', a length of time ''t'' where <code>-838:59:59</code> ≤ t ≤ <code>838:59:59</code> (note: you cannot store time to a precision greater than 1 second)
* '''YEAR''', a year ''y'' where 1901 ≤ y ≤ 2155

Note that '''TIMESTAMP''' is a highly versatile data type that can automatically change whenever a row is updated. For more information, refer to [http://dev.mysql.com/doc/refman/5.6/en/timestamp-initialization.html the MySQL documentation].

==== Field Options ====

By default, any "cell" in a MySQL table can contain the ''NULL'' value. If a certain field in your database schema will never contain a null value, you should declare it as '''NOT NULL''' in order to make queries more efficient.

You can also specify a default value for a field. If you do, when you insert data into the table, it won't be necessary to specify a value for that column. To specify a default value for a field, use the syntax '''DEFAULT 'DEFAULT-VALUE' '''

==== Create Table Example ====

The following query would create a table called ''employees'' with six columns: employee ID, first name, last name, nickname, department, and the date that they joined the company. Nickname is the only optional column. The primary key for the table (see the next section) will be the ''id'' column. The department will be one of five choices, with the default being CSE. The storage engine for the table will be InnoDB, and the default charset for the table will be UTF8.

<source lang="mysql">
create table employees (
id mediumint unsigned not null auto_increment,
first_name varchar(30) not null,
last_name varchar(40) not null,
nickname varchar(20),
department enum('CSE','BME','EECE','ESE','MEMS') not null default 'CSE',
joined timestamp not null default current_timestamp,
primary key (id)
) engine = InnoDB default character set = utf8 collate = utf8_general_ci;
</source>

=== Keys ===

When MySQL goes to get data out of a database, it does so using an index, like the index of a textbook. In tech talk, we call indices ''keys''.

==== Primary Key ====

The '''primary key''' is what MySQL uses to distinguish one row from another. Every table must have exactly one primary key; no more, no less.

It is common practice to add a column to a table with data type '''mediumint unsigned not null auto_increment''' to serve as the primary key for a table. The '''auto_increment''' is a neat little feature of MySQL that basically says, "when a new row is inserted into this table, assign it the next-lowest available ID".

To specify a primary key for a table, use the syntax:

<source lang="mysql">
primary key (column1, column2, column3)
</source>

==== Unique Key ====

If you want to specify that all items in a certain column will be unique from each other (like usernames or e-mail addresses), you can use a '''unique key'''.

Because a table can have more than one unique key, you need to give your unique key a name. It is common practice to name the key after the columns it is indexing, like ''idx_unique_column1_column2''.

The syntax for specifying a unique key is:

<source lang="mysql">
unique key <INDEX_NAME> (<column1>, <column2>, <column3>)
</source>

==== Foreign Key ====

A '''foreign key''' is the most complicated type of MySQL index. A foreign key says that the values in one column or columns are associated with the values in a column or columns in a ''different table''.

For example, suppose you had two tables, one containing users and the other containing posts on a forum. There is a one-to-many relationship between users and posts. You could link a post to a user by having a column in your post table called ''user_id'', which would be a ''foreign key'' to the ''id'' column in your users table.

Foreign keys can be a source of great confusion in developing database schemas. If you get stuck, keep this tip in mind: '''Foreign Keys should always reference the columns used in the Primary Key of a different table.'''

The syntax for creating a foreign key is as follows:

<source lang="mysql">
foreign key (<column1>, <column2>, <column3>) references <FOREIGN_TABLENAME> (<column1>, <column2>, <column3>)
</source>

For example, in the user-and-post example above, the foreign key declaration would be:

<source lang="mysql">
foreign key (user_id) references user (id)
</source>

===== Further Reading about Foreign Keys =====

* [[wikipedia:Foreign Key|Wikipedia Entry on Foreign Keys]]
* [http://www.techrepublic.com/article/an-introduction-to-foreign-keys-and-referential-integrity-in-mysql/6035435 An introductory article on Tech Republic]
* [http://dev.mysql.com/doc/refman/5.1/en/innodb-foreign-key-constraints.html MySQL Documentation for Foreign Keys]

==== Syntax for Keys ====

You can define keys in your '''create table''' query, or you can define them later.

If you define them in your '''create table''' query, do so after the column definitions but still inside the parentheses:

<source lang="mysql">
create table <TABLENAME> (
<column>,
<column>,
....,
primary key (<column1>, <column2>, ...),
unique key <INDEXNAME> (<column1>, <column2>, ...),
foreign key (<column1>, <column2>, ...) references <FOREIGN_TABLENAME> (<column1>, <column2>, ...)
)
</source>

To define indices later, use an '''ALTER TABLE''' query:

<source lang="mysql">
alter table <TABLENAME> add unique key <INDEXNAME> (<column1>, <column2>, ...);
alter table <TABLENAME> add foreign key (<column1>, <column2>, ...) references <FOREIGN_TABLENAME> (<column1>, <column2>, ...);
</source>

== State ==

The ''state'' of your database refers to the data that it holds. Whenever you insert, update, or delete data, the state of your database changes.

=== Inserting Data from a Query ===

To insert a row (aka ''entry'') into a table, use a query of the form

<source lang="mysql">
insert into TABLENAME (column1, column2, column3) values (value1_1, value2_1, value3_1), (value1_2, value2_2, value3_2), ...
</source>

For example, to insert an employee into the table we created earlier, you could use the query

<source lang="mysql">
insert into employees (first_name, last_name, department) values ('John', 'Doe', 'BME'), ('Keith', 'Jones', 'ESE');
</source>

Because of the various default and auto_increment values we have specified in our database, the database would now contain the following data:

+----+------------+-----------+----------+------------+---------------------+
| id | first_name | last_name | nickname | department | joined |
+----+------------+-----------+----------+------------+---------------------+
| 1 | John | Doe | NULL | BME | 2012-08-22 22:02:24 |
| 2 | Keith | Jones | NULL | ESE | 2012-08-22 22:02:24 |
+----+------------+-----------+----------+------------+---------------------+

If you want to insert a DATE or DATETIME value into a database, use the YYYY-MM-DD HH:MM:SS format. The exception is with TIMESTAMP, in which you should use a 16-bit integer representing the number of seconds since January 1, 1970.

=== Inserting Data from a File ===

The '''load data infile''' command populates a table using data stored in a text file. The general form of the command is:

<source lang="mysql">
LOAD DATA INFILE 'dataFile.txt' INTO TABLE table_name;
</source>

where ''dataFile.txt'' is located relative to the server's data directory if a path name with leading components is given; otherwise, if no leading components are given, the server assumes the file is in the default database's database directory. If an absolute path is specified, the server uses it. For this command to work, you must issue the following command to grant the FILE privilege to your user:

<source lang="mysql">
GRANT FILE on *.* to 'user'@'host';
</source>

Alternatively, and quite usefully, you can load data from a local file (a file residing on the same machine as the client) using the LOCAL keyword. When loading data from the local machine you do not need to grant your user the FILE privilege, think about why this might be the case. (Hint: what is going on behind the scenes when you load the data in from a local machine when your database is hosted on a different machine?)

<source lang="mysql">
LOAD DATA LOCAL INFILE '/home/linus_torvalds/Documents/dataFile.txt' INTO TABLE table_name;
</source>

'''Note:''' When the local keyword is used, it is best to specify an absolute path.

'''Additional Note:''' If we SSH into our EC2 instance and then run the mysql command, the client is the EC2 instance, not the local computer that we SSH from. The LOCAL keyword in this case is allowing us to load files from the EC2 instance, not the local computer that we are using.

By default, the text files are assumed to have tab-delimited columns with column values specified in the same order as they are defined in the table (though you can specify additional/alternative delimiters using additional options).

=== Updating Data ===

To update a row in a table, use a query of the form

<source lang="mysql">
update TABLENAME set column1=value1, column2=value2, column3=value3 where CONDITION
</source>

For example, to set Keith Jones's nickname to "KJ", you would run the query

<source lang="mysql">
update employees set nickname='KJ' where id=2;
</source>

=== Deleting Data ===

To delete a row from a table, use a query of the form

<source lang="mysql">
delete from TABLENAME where CONDITION
</source>

For example, to delete John Doe from the table, you would run the query

<source lang="mysql">
delete from employees where id=1;
</source>

'''DO NOT FORGET THE WHERE CLAUSE!''' If you do, ''all'' rows will be deleted from your table!

== Selecting Data ==

The most simple query to select data from a table is:

<source lang="mysql">
select * from TABLENAME;
</source>

However, in practice, you will almost always want to perform a more advanced query.

=== Selecting Specific Columns ===

In order to save transmission time, you should select only those columns that you need in your application. The syntax for selecting only certain columns is

<source lang="mysql">
select column1, column2, column3 from TABLENAME;
</source>

=== Computation ===

MySQL can perform computation. For example, if you wanted to select the amount of time that a certain employee has been at the company, you could run the query

<source lang="mysql">
select TIMESTAMPDIFF( second, joined, NOW() ) as time_with_company from employees;
</source>

=== Joins ===

When you have one-to-one, one-to-many, or many-to-many relationships between tables, you will often want to select data from multiple tables in one query. To do this, use '''joins'''.

Joins are best described by example. Suppose you had the following tables in a MySQL database:

mysql> describe employees;
+------------+---------------------------------------+------+-----+-------------------+----------------+
| Field | Type | Null | Key | Default | Extra |
+------------+---------------------------------------+------+-----+-------------------+----------------+
| id | mediumint(8) unsigned | NO | PRI | NULL | auto_increment |
| first_name | varchar(30) | NO | | NULL | |
| last_name | varchar(40) | NO | | NULL | |
| nickname | varchar(20) | YES | | NULL | |
| department | enum('CSE','BME','EECE','ESE','MEMS') | NO | | CSE | |
| joined | timestamp | NO | | CURRENT_TIMESTAMP | |
+------------+---------------------------------------+------+-----+-------------------+----------------+
6 rows in set (0.00 sec)

mysql> describe projects;
+--------------+-----------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+--------------+-----------------------+------+-----+---------+----------------+
| id | mediumint(8) unsigned | NO | PRI | NULL | auto_increment |
| employee_id | mediumint(8) unsigned | NO | MUL | NULL | |
| project_name | tinytext | NO | | NULL | |
+--------------+-----------------------+------+-----+---------+----------------+
3 rows in set (0.00 sec)

Further, suppose the tables had the following data:

mysql> select * from employees;
+----+------------+-----------+----------+------------+---------------------+
| id | first_name | last_name | nickname | department | joined |
+----+------------+-----------+----------+------------+---------------------+
| 1 | John | Doe | NULL | BME | 2012-08-22 22:02:24 |
| 2 | Keith | Jones | KK | ESE | 2012-08-22 22:02:24 |
+----+------------+-----------+----------+------------+---------------------+
2 rows in set (0.00 sec)

mysql> select * from projects;
+----+-------------+-----------------------------------+
| id | employee_id | project_name |
+----+-------------+-----------------------------------+
| 1 | 1 | Interview the Chancellor |
| 2 | 1 | Give Presentation in Mumbai |
| 3 | 2 | Publish Research Paper in Journal |
+----+-------------+-----------------------------------+
3 rows in set (0.00 sec)

If you ran the following query:

<source lang="mysql">
select
project_name,
employees.first_name,
employees.last_name
from projects
join employees on (projects.employee_id=employees.id);
</source>

You would get the result:

mysql> select project_name, employees.first_name, employees.last_name from projects
-> join employees on (projects.employee_id=employees.id);
+-----------------------------------+------------+-----------+
| project_name | first_name | last_name |
+-----------------------------------+------------+-----------+
| Interview the Chancellor | John | Doe |
| Give Presentation in Mumbai | John | Doe |
| Publish Research Paper in Journal | Keith | Jones |
+-----------------------------------+------------+-----------+
3 rows in set (0.00 sec)

Isn't that cool?

==== Inner Joins, Left Joins, Right Joins, and Full Joins ====

When you perform a '''JOIN''' like we did above, you are implicitly performing an '''INNER JOIN'''. However, there may be times when you want to select data in a query even if it doesn't have a complement in the joined table. For example, suppose Keith Jones cancelled his project. The projects table now contains the following data:

mysql> select * from projects;
+----+-------------+-----------------------------+
| id | employee_id | project_name |
+----+-------------+-----------------------------+
| 1 | 1 | Interview the Chancellor |
| 2 | 1 | Give Presentation in Mumbai |
+----+-------------+-----------------------------+
2 rows in set (0.00 sec)

Now, we could run the following query:

<source lang="mysql">
select
count(projects.id) as project_num,
employees.first_name,
employees.last_name
from employees
join projects on (employees.id=projects.employee_id)
group by employees.id;
</source>

We get the result:

mysql> select count(projects.id) as project_num, employees.first_name, employees.last_name from employees
-> join projects on (employees.id=projects.employee_id) group by employees.id;
+-------------+------------+-----------+
| project_num | first_name | last_name |
+-------------+------------+-----------+
| 2 | John | Doe |
+-------------+------------+-----------+
1 row in set (0.00 sec)

But where's Keith Jones? Since Keith Jones didn't have any projects, he was omitted from the result. This is probably not what we intended to happen. Instead, you should perform a '''LEFT JOIN''', like so:

<source lang="mysql">
select
count(projects.id) as project_num,
employees.first_name,
employees.last_name
from employees
left join projects on (employees.id=projects.employee_id)
group by employees.id;
</source>

The result will now be:

mysql> select count(projects.id) as project_num, employees.first_name, employees.last_name from employees
-> left join projects on (employees.id=projects.employee_id) group by employees.id;
+-------------+------------+-----------+
| project_num | first_name | last_name |
+-------------+------------+-----------+
| 2 | John | Doe |
| 0 | Keith | Jones |
+-------------+------------+-----------+
2 rows in set (0.00 sec)

(For more information on COUNT and GROUP BY, see [[#Aggregation Clauses|a later section]].)

You can see that a LEFT JOIN preserves all rows from the "left" table in the join. Likewise, a RIGHT JOIN preserves all rows from the "right" table. A FULL JOIN preserves all rows from both tables.

=== WHERE Clauses ===

If you want to select data that satisfies a condition, use a '''WHERE''' clause.

A simple '''WHERE''' clause can use equality, less than, greater than, and most other comparison operators. For example, the following query selects all last names of employees with an Employee ID of less than 10:

<source lang="mysql">
select
last_name
from employees
where
employee_id < 10;
</source>

If we wanted to select everyone from our employees table whose name started with a J, we could use a '''LIKE''' statement in our '''WHERE''' clause:

<source lang="mysql">
select
first_name,
last_name
from employees
where
first_name like 'J%';
</source>

The '''LIKE''' clause is SQL's way of letting you perform simple tests on strings. (SQL's built-in string comparison operations are not as powerful as regular expressions, but they'll get the job done in most everyday situations.)

=== Sorting Data ===

You can sort data using an '''ORDER BY''' clause like so:

<source lang="mysql">
select
first_name,
last_name
from employees
order by last_name asc;
</source>

If you wanted to select only the first few results from a query, use the '''LIMIT''' clause:

<source lang="mysql">
select
first_name,
last_name
from employees
order by last_name asc
limit 10;
</source>

It is important to note that the ORDER BY on a field is performed ''after'' aggregation functions. Currently, there is no way in MySQL to perform sorting ''before'' aggregation has occurred in a single query; we need to resort to nested queries. (For example, in the baseball example below, it's not easy to select "the player from each team with the highest batting average". It's even harder, if not impossible, to select "the ''three'' players from each team with the highest batting averages".)

=== Aggregation Clauses ===

When performing computations on an entire database, you will probably find yourself wanting to use ''aggregation functions''. Aggregation functions include:

* '''AVG(column_name)''' to compute the average value of column_name
* '''STD(column_name)''' to compute the standard deviation of column_name
* '''SUM(column_name)''' to compute the sum
* '''MIN(column_name)''' to find the minimum value
* '''MAX(column_name)''' to fund the maximum value
* '''COUNT(column_name)''' to count the total number of entries having a not null entry in column_name

For example, if you had a table containing batter averages, you could calculate the mean batter average using this query:

<source lang="mysql">
select
avg(batter_average) as avg_batter_average
from baseball_players;
</source>

Since WHERE clauses are performed ''before'' aggregation, you can also perform a query like

<source lang="mysql">
select
avg(batter_average) as avg_batter_average
from baseball_players
where
team_id=5;
</source>

If you want to aggregate all rows that share a certain value in an index, use a '''GROUP BY''' clause. For example, to select the average batter average for all teams, you could do

<source lang="mysql">
select
avg(batter_average) as avg_batter_average
from baseball_players
group by team_id;
</source>

You may find yourself wanting to apply a condition after the aggregation has occurred. You can do this using a '''HAVING''' clause. For example, the following query would select all baseball teams whose average batter average is more than 300:

<source lang="mysql">
select
avg(batter_average) as avg_batter_average
from baseball_players
group by team_id
having
avg_batter_average > 300;
</source>

It is perfectly valid to combine WHERE and HAVING clauses in a single query. For example, the following query first finds all players whose first name starts with "J", averages their batting averages together by team, and then selects those teams whose average batting average of the "J" players is more than 300; that is, it selects those teams "whose average batting average of players whose first name starts with J is more than 300":

<source lang="mysql">
select
avg(batter_average) as avg_batter_average
from baseball_players
where
first_name like "J%"
group by team_id
having
avg_batter_average > 300;
</source>

[[Category:Module 3]]

MySQL Schema and State

2023-01-10T06:00:02Z

Zach: /* Foreign Key */

MySQL is a powerful database that allows for complex logic when manipulating data. This guide is an introduction and a reference to common SQL queries for schema and state manipulation.

This guide assumes that you already have a working copy of MySQL. If you don't, read [[Introduction to MySQL]] first.

==Schema ==

The ''schema'' of your database refers to the way that your data is structured and to how bits of data relate to each other. Every database in your MySQL server has a separate schema, and separate databases should not typically interact with one another.

=== Tables ===

A database contains one or more ''tables''. Tables contain columns and rows, which you may sometimes call ''fields'' and ''entries''.

You can see all of the tables in the database using this query:

<source lang="mysql">
show tables;
</source>

When you create a table, you should provide information about all of that table's columns within the ''create table'' query. Here is the general syntax for the query:

<source lang="mysql">
create table <TABLENAME> (
<FIELDNAME> <DATATYPE> <OPTIONS>,
<FIELDNAME> <DATATYPE> <OPTIONS>,
<FIELDNAME> <DATATYPE> <OPTIONS>,
<FIELDNAME> <DATATYPE> <OPTIONS>
) <TABLE_OPTIONS>;
</source>

You can get details about an existing table using this query:

<source lang="mysql">
describe <TABLENAME>;
</source>

==== Data Types ====

Just like a programming language, MySQL has data types. There are dozens of data types; some of the most common ones are documented below.

===== String Data Types =====

There are four main types of string data types: character, text, blob, and enumerable.

* Character types are useful for short strings, like usernames and e-mail addresses.
* Text types differ from character types in that behind the scenes, they are not actually stored with the rest of the table row; they are useful for longer blocks of text, like forum posts.
* Blobs are for binary data, like files.
* Enumerables are for something that could have one of a set number of values (not more than 65535), like what State a user is from (Alabama, Alaska, and so on). Enums are also useful for storing boolean data, like '''ENUM('yes','no')'''. An enum is actually stored as an integer in the database, making queries more efficient.

The syntax for these data types is:

* '''VARCHAR(#)''', a string ''at most'' # characters long, where 0 ≤ # ≤ 65535
* '''CHAR(#)''', a string ''exactly'' # characters long, where 0 ≤ # ≤ 65535
* '''TINYTEXT''', a text block at most 255 characters long
* '''TEXT''', a text block at most 65535 characters long
* '''LONGTEXT''', a text block at most 4 GB in length
* '''BLOB''', a binary string at most 65 KB in length
* '''MEDIUMBLOB''', a binary string at most 16 MB in length
* '''LONGBLOB''', a binary string at most 4 GB in length
* '''ENUM('a', 'b', 'c')''', an enumerable with choices a, b, and c

===== Numeric Data Types =====

There are two types of numeric data types that you need to worry about: integers and decimals. The difference between the two should be pretty self-explanatory. The syntax for using these data types is:

* '''TINYINT''', an integer ''n'' where -128 ≤ n ≤ 127
* '''TINYINT UNSIGNED''', an integer ''n'' where 0 ≤ n ≤ 255
* '''SMALLINT''', an integer ''n'' where -32,768 ≤ n ≤ 32,767
* '''SMALLINT UNSIGNED''', an integer ''n'' where 0 ≤ n ≤ 65,535
* '''MEDIUMINT''', an integer ''n'' where -8,388,608 ≤ n ≤ 8,388,607
* '''MEDIUMINT UNSIGNED''', an integer ''n'' where 0 ≤ n ≤ 16,777,215
* '''INT''', an integer ''n'' where -2,147,483,648 ≤ n ≤ 2,147,483,647
* '''INT UNSIGNED''', an integer ''n'' where 0 ≤ n ≤ 4,294,967,295
* '''BIGINT''', an integer ''n'' where -9,223,372,036,854,775,808 ≤ n ≤ 9,223,372,036,854,775,807
* '''BIGINT UNSIGNED''', an integer ''n'' where 0 ≤ n ≤ 18,446,744,073,709,551,615
* '''DECIMAL(#a, #b)''', a decimal number at most ''#a'' digits long, ''#b'' of which are after the decimal point. For example, ''DECIMAL(5,2)'' can contain ''n'' where -999.99 ≤ n ≤ 999.99

When choosing the size of an integer, you should use the ''smallest integer for which you know you will never "run out of room"''. In most everyday circumstances, '''MEDIUMINT''' should be the biggest you'll need.

===== Temporal Data Types =====
{{XKCD
|name=2038
|id=607
}}

MySQL provides a class of data types that specialize in handling dates and times. The syntax for using date types is:

* '''DATE''', a date ''d'' where <code>1000-01-01</code> ≤ d ≤ <code>9999-12-31</code>
* '''DATETIME''', a date with time ''d'' where <code>1000-01-01 00:00:00</code> ≤ d ≤ <code>9999-12-31 23:59:59</code>
* '''TIMESTAMP''', a date with time ''d'' where <code>1970-01-01 00:00:01 UTC</code> ≤ d ≤ <code>2038-01-19 03:14:07 UTC</code> (note: <code>1970-01-01 00:00:01 UTC</code> is the Unix epoch)
* '''TIME''', a length of time ''t'' where <code>-838:59:59</code> ≤ t ≤ <code>838:59:59</code> (note: you cannot store time to a precision greater than 1 second)
* '''YEAR''', a year ''y'' where 1901 ≤ y ≤ 2155

Note that '''TIMESTAMP''' is a highly versatile data type that can automatically change whenever a row is updated. For more information, refer to [http://dev.mysql.com/doc/refman/5.6/en/timestamp-initialization.html the MySQL documentation].

==== Field Options ====

By default, any "cell" in a MySQL table can contain the ''NULL'' value. If a certain field in your database schema will never contain a null value, you should declare it as '''NOT NULL''' in order to make queries more efficient.

You can also specify a default value for a field. If you do, when you insert data into the table, it won't be necessary to specify a value for that column. To specify a default value for a field, use the syntax '''DEFAULT 'DEFAULT-VALUE' '''

==== Create Table Example ====

The following query would create a table called ''employees'' with six columns: employee ID, first name, last name, nickname, department, and the date that they joined the company. Nickname is the only optional column. The primary key for the table (see the next section) will be the ''id'' column. The department will be one of five choices, with the default being CSE. The storage engine for the table will be InnoDB, and the default charset for the table will be UTF8.

<source lang="mysql">
create table employees (
id mediumint unsigned not null auto_increment,
first_name varchar(30) not null,
last_name varchar(40) not null,
nickname varchar(20),
department enum('CSE','BME','EECE','ESE','MEMS') not null default 'CSE',
joined timestamp not null default current_timestamp,
primary key (id)
) engine = InnoDB default character set = utf8 collate = utf8_general_ci;
</source>

=== Keys ===

When MySQL goes to get data out of a database, it does so using an index, like the index of a textbook. In tech talk, we call indices ''keys''.

==== Primary Key ====

The '''primary key''' is what MySQL uses to distinguish one row from another. Every table must have exactly one primary key; no more, no less.

It is common practice to add a column to a table with data type '''mediumint unsigned not null auto_increment''' to serve as the primary key for a table. The '''auto_increment''' is a neat little feature of MySQL that basically says, "when a new row is inserted into this table, assign it the next-lowest available ID".

To specify a primary key for a table, use the syntax:

<source lang="mysql">
primary key (column1, column2, column3)
</source>

==== Unique Key ====

If you want to specify that all items in a certain column will be unique from each other (like usernames or e-mail addresses), you can use a '''unique key'''.

Because a table can have more than one unique key, you need to give your unique key a name. It is common practice to name the key after the columns it is indexing, like ''idx_unique_column1_column2''.

The syntax for specifying a unique key is:

<source lang="mysql">
unique key <INDEX_NAME> (<column1>, <column2>, <column3>)
</source>

==== Foreign Key ====

A '''foreign key''' is the most complicated type of MySQL index. A foreign key says that the values in one column or columns are associated with the values in a column or columns in a ''different table''.

For example, suppose you had two tables, one containing users and the other containing posts on a forum. There is a one-to-many relationship between users and posts. You could link a post to a user by having a column in your post table called ''user_id'', which would be a ''foreign key'' to the ''id'' column in your users table.

Foreign keys can be a source of great confusion in developing database schemas. If you get stuck, keep this tip in mind: '''Foreign Keys should always reference the columns used in the Primary Key of a different table.'''

The syntax for creating a foreign key is as follows:

<source lang="mysql">
foreign key (<column1>, <column2>, <column3>) references <FOREIGN_TABLENAME> (<column1>, <column2>, <column3>)
</source>

For example, in the user-and-post example above, the foreign key declaration would be:

<source lang="mysql">
foreign key (user_id) references user (id)
</source>

===== Further Reading about Foreign Keys =====

* [[wikipedia:Foreign Key|Wikipedia Entry on Foreign Keys]]
* [http://www.techrepublic.com/article/an-introduction-to-foreign-keys-and-referential-integrity-in-mysql/6035435 An introductory article on Tech Republic]
* [http://dev.mysql.com/doc/refman/5.1/en/innodb-foreign-key-constraints.html MySQL Documentation for Foreign Keys]

==== Syntax for Keys ====

You can define keys in your '''create table''' query, or you can define them later.

If you define them in your '''create table''' query, do so after the column definitions but still inside the parentheses:

<source lang="mysql">
create table TABLENAME (
column,
column,
....,
primary key (column1, column2, ...),
unique key INDEXNAME (column1, column2, ...),
foreign key (column1, column2, ...) references FOREIGN_TABLENAME (column1, column2, ...)
)
</source>

To define indices later, use an '''ALTER TABLE''' query:

<source lang="mysql">
alter table TABLENAME add unique key INDEXNAME (column1, column2, ...);
alter table TABLENAME add foreign key (column1, column2, ...) references FOREIGN_TABLENAME (column1, column2, ...);
</source>

== State ==

The ''state'' of your database refers to the data that it holds. Whenever you insert, update, or delete data, the state of your database changes.

=== Inserting Data from a Query ===

To insert a row (aka ''entry'') into a table, use a query of the form

<source lang="mysql">
insert into TABLENAME (column1, column2, column3) values (value1_1, value2_1, value3_1), (value1_2, value2_2, value3_2), ...
</source>

For example, to insert an employee into the table we created earlier, you could use the query

<source lang="mysql">
insert into employees (first_name, last_name, department) values ('John', 'Doe', 'BME'), ('Keith', 'Jones', 'ESE');
</source>

Because of the various default and auto_increment values we have specified in our database, the database would now contain the following data:

+----+------------+-----------+----------+------------+---------------------+
| id | first_name | last_name | nickname | department | joined |
+----+------------+-----------+----------+------------+---------------------+
| 1 | John | Doe | NULL | BME | 2012-08-22 22:02:24 |
| 2 | Keith | Jones | NULL | ESE | 2012-08-22 22:02:24 |
+----+------------+-----------+----------+------------+---------------------+

If you want to insert a DATE or DATETIME value into a database, use the YYYY-MM-DD HH:MM:SS format. The exception is with TIMESTAMP, in which you should use a 16-bit integer representing the number of seconds since January 1, 1970.

=== Inserting Data from a File ===

The '''load data infile''' command populates a table using data stored in a text file. The general form of the command is:

<source lang="mysql">
LOAD DATA INFILE 'dataFile.txt' INTO TABLE table_name;
</source>

where ''dataFile.txt'' is located relative to the server's data directory if a path name with leading components is given; otherwise, if no leading components are given, the server assumes the file is in the default database's database directory. If an absolute path is specified, the server uses it. For this command to work, you must issue the following command to grant the FILE privilege to your user:

<source lang="mysql">
GRANT FILE on *.* to 'user'@'host';
</source>

Alternatively, and quite usefully, you can load data from a local file (a file residing on the same machine as the client) using the LOCAL keyword. When loading data from the local machine you do not need to grant your user the FILE privilege, think about why this might be the case. (Hint: what is going on behind the scenes when you load the data in from a local machine when your database is hosted on a different machine?)

<source lang="mysql">
LOAD DATA LOCAL INFILE '/home/linus_torvalds/Documents/dataFile.txt' INTO TABLE table_name;
</source>

'''Note:''' When the local keyword is used, it is best to specify an absolute path.

'''Additional Note:''' If we SSH into our EC2 instance and then run the mysql command, the client is the EC2 instance, not the local computer that we SSH from. The LOCAL keyword in this case is allowing us to load files from the EC2 instance, not the local computer that we are using.

By default, the text files are assumed to have tab-delimited columns with column values specified in the same order as they are defined in the table (though you can specify additional/alternative delimiters using additional options).

=== Updating Data ===

To update a row in a table, use a query of the form

<source lang="mysql">
update TABLENAME set column1=value1, column2=value2, column3=value3 where CONDITION
</source>

For example, to set Keith Jones's nickname to "KJ", you would run the query

<source lang="mysql">
update employees set nickname='KJ' where id=2;
</source>

=== Deleting Data ===

To delete a row from a table, use a query of the form

<source lang="mysql">
delete from TABLENAME where CONDITION
</source>

For example, to delete John Doe from the table, you would run the query

<source lang="mysql">
delete from employees where id=1;
</source>

'''DO NOT FORGET THE WHERE CLAUSE!''' If you do, ''all'' rows will be deleted from your table!

== Selecting Data ==

The most simple query to select data from a table is:

<source lang="mysql">
select * from TABLENAME;
</source>

However, in practice, you will almost always want to perform a more advanced query.

=== Selecting Specific Columns ===

In order to save transmission time, you should select only those columns that you need in your application. The syntax for selecting only certain columns is

<source lang="mysql">
select column1, column2, column3 from TABLENAME;
</source>

=== Computation ===

MySQL can perform computation. For example, if you wanted to select the amount of time that a certain employee has been at the company, you could run the query

<source lang="mysql">
select TIMESTAMPDIFF( second, joined, NOW() ) as time_with_company from employees;
</source>

=== Joins ===

When you have one-to-one, one-to-many, or many-to-many relationships between tables, you will often want to select data from multiple tables in one query. To do this, use '''joins'''.

Joins are best described by example. Suppose you had the following tables in a MySQL database:

mysql> describe employees;
+------------+---------------------------------------+------+-----+-------------------+----------------+
| Field | Type | Null | Key | Default | Extra |
+------------+---------------------------------------+------+-----+-------------------+----------------+
| id | mediumint(8) unsigned | NO | PRI | NULL | auto_increment |
| first_name | varchar(30) | NO | | NULL | |
| last_name | varchar(40) | NO | | NULL | |
| nickname | varchar(20) | YES | | NULL | |
| department | enum('CSE','BME','EECE','ESE','MEMS') | NO | | CSE | |
| joined | timestamp | NO | | CURRENT_TIMESTAMP | |
+------------+---------------------------------------+------+-----+-------------------+----------------+
6 rows in set (0.00 sec)

mysql> describe projects;
+--------------+-----------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+--------------+-----------------------+------+-----+---------+----------------+
| id | mediumint(8) unsigned | NO | PRI | NULL | auto_increment |
| employee_id | mediumint(8) unsigned | NO | MUL | NULL | |
| project_name | tinytext | NO | | NULL | |
+--------------+-----------------------+------+-----+---------+----------------+
3 rows in set (0.00 sec)

Further, suppose the tables had the following data:

mysql> select * from employees;
+----+------------+-----------+----------+------------+---------------------+
| id | first_name | last_name | nickname | department | joined |
+----+------------+-----------+----------+------------+---------------------+
| 1 | John | Doe | NULL | BME | 2012-08-22 22:02:24 |
| 2 | Keith | Jones | KK | ESE | 2012-08-22 22:02:24 |
+----+------------+-----------+----------+------------+---------------------+
2 rows in set (0.00 sec)

mysql> select * from projects;
+----+-------------+-----------------------------------+
| id | employee_id | project_name |
+----+-------------+-----------------------------------+
| 1 | 1 | Interview the Chancellor |
| 2 | 1 | Give Presentation in Mumbai |
| 3 | 2 | Publish Research Paper in Journal |
+----+-------------+-----------------------------------+
3 rows in set (0.00 sec)

If you ran the following query:

<source lang="mysql">
select
project_name,
employees.first_name,
employees.last_name
from projects
join employees on (projects.employee_id=employees.id);
</source>

You would get the result:

mysql> select project_name, employees.first_name, employees.last_name from projects
-> join employees on (projects.employee_id=employees.id);
+-----------------------------------+------------+-----------+
| project_name | first_name | last_name |
+-----------------------------------+------------+-----------+
| Interview the Chancellor | John | Doe |
| Give Presentation in Mumbai | John | Doe |
| Publish Research Paper in Journal | Keith | Jones |
+-----------------------------------+------------+-----------+
3 rows in set (0.00 sec)

Isn't that cool?

==== Inner Joins, Left Joins, Right Joins, and Full Joins ====

When you perform a '''JOIN''' like we did above, you are implicitly performing an '''INNER JOIN'''. However, there may be times when you want to select data in a query even if it doesn't have a complement in the joined table. For example, suppose Keith Jones cancelled his project. The projects table now contains the following data:

mysql> select * from projects;
+----+-------------+-----------------------------+
| id | employee_id | project_name |
+----+-------------+-----------------------------+
| 1 | 1 | Interview the Chancellor |
| 2 | 1 | Give Presentation in Mumbai |
+----+-------------+-----------------------------+
2 rows in set (0.00 sec)

Now, we could run the following query:

<source lang="mysql">
select
count(projects.id) as project_num,
employees.first_name,
employees.last_name
from employees
join projects on (employees.id=projects.employee_id)
group by employees.id;
</source>

We get the result:

mysql> select count(projects.id) as project_num, employees.first_name, employees.last_name from employees
-> join projects on (employees.id=projects.employee_id) group by employees.id;
+-------------+------------+-----------+
| project_num | first_name | last_name |
+-------------+------------+-----------+
| 2 | John | Doe |
+-------------+------------+-----------+
1 row in set (0.00 sec)

But where's Keith Jones? Since Keith Jones didn't have any projects, he was omitted from the result. This is probably not what we intended to happen. Instead, you should perform a '''LEFT JOIN''', like so:

<source lang="mysql">
select
count(projects.id) as project_num,
employees.first_name,
employees.last_name
from employees
left join projects on (employees.id=projects.employee_id)
group by employees.id;
</source>

The result will now be:

mysql> select count(projects.id) as project_num, employees.first_name, employees.last_name from employees
-> left join projects on (employees.id=projects.employee_id) group by employees.id;
+-------------+------------+-----------+
| project_num | first_name | last_name |
+-------------+------------+-----------+
| 2 | John | Doe |
| 0 | Keith | Jones |
+-------------+------------+-----------+
2 rows in set (0.00 sec)

(For more information on COUNT and GROUP BY, see [[#Aggregation Clauses|a later section]].)

You can see that a LEFT JOIN preserves all rows from the "left" table in the join. Likewise, a RIGHT JOIN preserves all rows from the "right" table. A FULL JOIN preserves all rows from both tables.

=== WHERE Clauses ===

If you want to select data that satisfies a condition, use a '''WHERE''' clause.

A simple '''WHERE''' clause can use equality, less than, greater than, and most other comparison operators. For example, the following query selects all last names of employees with an Employee ID of less than 10:

<source lang="mysql">
select
last_name
from employees
where
employee_id < 10;
</source>

If we wanted to select everyone from our employees table whose name started with a J, we could use a '''LIKE''' statement in our '''WHERE''' clause:

<source lang="mysql">
select
first_name,
last_name
from employees
where
first_name like 'J%';
</source>

The '''LIKE''' clause is SQL's way of letting you perform simple tests on strings. (SQL's built-in string comparison operations are not as powerful as regular expressions, but they'll get the job done in most everyday situations.)

=== Sorting Data ===

You can sort data using an '''ORDER BY''' clause like so:

<source lang="mysql">
select
first_name,
last_name
from employees
order by last_name asc;
</source>

If you wanted to select only the first few results from a query, use the '''LIMIT''' clause:

<source lang="mysql">
select
first_name,
last_name
from employees
order by last_name asc
limit 10;
</source>

It is important to note that the ORDER BY on a field is performed ''after'' aggregation functions. Currently, there is no way in MySQL to perform sorting ''before'' aggregation has occurred in a single query; we need to resort to nested queries. (For example, in the baseball example below, it's not easy to select "the player from each team with the highest batting average". It's even harder, if not impossible, to select "the ''three'' players from each team with the highest batting averages".)

=== Aggregation Clauses ===

When performing computations on an entire database, you will probably find yourself wanting to use ''aggregation functions''. Aggregation functions include:

* '''AVG(column_name)''' to compute the average value of column_name
* '''STD(column_name)''' to compute the standard deviation of column_name
* '''SUM(column_name)''' to compute the sum
* '''MIN(column_name)''' to find the minimum value
* '''MAX(column_name)''' to fund the maximum value
* '''COUNT(column_name)''' to count the total number of entries having a not null entry in column_name

For example, if you had a table containing batter averages, you could calculate the mean batter average using this query:

<source lang="mysql">
select
avg(batter_average) as avg_batter_average
from baseball_players;
</source>

Since WHERE clauses are performed ''before'' aggregation, you can also perform a query like

<source lang="mysql">
select
avg(batter_average) as avg_batter_average
from baseball_players
where
team_id=5;
</source>

If you want to aggregate all rows that share a certain value in an index, use a '''GROUP BY''' clause. For example, to select the average batter average for all teams, you could do

<source lang="mysql">
select
avg(batter_average) as avg_batter_average
from baseball_players
group by team_id;
</source>

You may find yourself wanting to apply a condition after the aggregation has occurred. You can do this using a '''HAVING''' clause. For example, the following query would select all baseball teams whose average batter average is more than 300:

<source lang="mysql">
select
avg(batter_average) as avg_batter_average
from baseball_players
group by team_id
having
avg_batter_average > 300;
</source>

It is perfectly valid to combine WHERE and HAVING clauses in a single query. For example, the following query first finds all players whose first name starts with "J", averages their batting averages together by team, and then selects those teams whose average batting average of the "J" players is more than 300; that is, it selects those teams "whose average batting average of players whose first name starts with J is more than 300":

<source lang="mysql">
select
avg(batter_average) as avg_batter_average
from baseball_players
where
first_name like "J%"
group by team_id
having
avg_batter_average > 300;
</source>

[[Category:Module 3]]