Difference between revisions of "Problem Solving Assignment: File Permissions"
| Line 57: | Line 57: | ||
To test your site, you need to do two things: | To test your site, you need to do two things: | ||
| − | # | + | #Set up Github Pages |
| − | # Submit a sandbox hit that you can try out | + | # Commit and push your code to Github. It may take a couple of minutes, but your changes will eventually be reflected in your Github pages site. |
| + | # Submit a sandbox hit that you can try out. ([[Submitting Sandbox HITs]]) | ||
# If you open your developer's console in your browser and see red errors, then something is broken and will be broken when you submit your hits. You must fix it. | # If you open your developer's console in your browser and see red errors, then something is broken and will be broken when you submit your hits. You must fix it. | ||
| − | == | + | ==Setting Up Pages== |
| − | To | + | To see what your version of the file permissions interface, you can view it through github pages. This will also be how mTurk users will view and interact with your site, so it's important to get going even if you plan to develop fully locally. |
| − | + | # From your github repository, go the the settings tab and scroll down to the bottom. | |
| + | # Change the repository visibility to 'public.' | ||
| + | # Enable github Pages. Select the 'main' branch and the 'root' directory to serve pages from. | ||
| + | # Once you hit save, github will add a link to your hosted pages so that you can see them. | ||
| + | ==Uploading to Github Pages== | ||
| + | To test or submit your project, you'll first need to upload your to Github. | ||
| + | |||
| + | # If you've made changes to javascript and/or typescript code, please rebuild your code to get the new changes. (see [[Node Setup]] - step 6) | ||
# Test your site locally via localhost (see [[Running Sites Locally]]) | # Test your site locally via localhost (see [[Running Sites Locally]]) | ||
# Commit and push your changes (see [https://code.visualstudio.com/docs/editor/versioncontrol Version Control in VS Code]) | # Commit and push your changes (see [https://code.visualstudio.com/docs/editor/versioncontrol Version Control in VS Code]) | ||
| − | + | # It will take a few minutes but your Github pages site will eventually reflect your changes. | |
| − | # It will take a | ||
| − | |||
==Making a Sandbox HIT== | ==Making a Sandbox HIT== | ||
If you want to see what your MTurk workers will see, you can submit a sandbox hit. | If you want to see what your MTurk workers will see, you can submit a sandbox hit. | ||
| − | # | + | # This page offers a walkthrough on how to do that. ([[Submitting Sandbox HITs]]) |
| − | |||
[[File:Screenshot (14).png|thumb|Worker Sandbox with relevant ui elements highlighted.]] | [[File:Screenshot (14).png|thumb|Worker Sandbox with relevant ui elements highlighted.]] | ||
Revision as of 02:49, 11 March 2021
Contents
Task Description
The goal of this assignment is to use a GenderMag analysis to improve a File Permissions interface that is highly similar to the one used in Windows 10. We've developed a series of tasks that combine different kinds of tasks and system states to evaluate the overall system effectiveness. For each scenario, you will see a simplified file system management interface. In this project, you users will be given a scenario and asked to modify the file permissions to achieve a goal presented in that scenario.
Specifically, we'll evaluate the following scenarios:
Easy
- Remove some permissions from employee (V1) (remove_direct_permission) Baseline = 0.96; Passing Score >= 1.15
- Add permissions for a new employee (add_new_user) Baseline = 1.27; Passing Score >= 1.52
- Add permissions for a manager (add_full_permissions) Baseline = 1.96; Passing Score >= 2.35
Medium
- Remove some permissions from employee (V2) (remove_inherited_permission) Baseline = 1.63; Passing Score >= 1.95
- Add permissions for an intern (intern_permissions) Baseline = 0.9; Passing Score >= 1.08
- Remove ALL permissions from employee (remove_user_with_inheritance) Baseline = 1.34; Passing Score >= 1.60
Hard
- Remove some permissions from employee (V3) (restrict_group_member) Baseline = 0.97; Passing Score >= 1.16
- Fix permissions for a teaching assistant (let_ta_modify) Baseline = 0; Passing Score >= 0.2
- Fix permissions for all users (wrapper_lost_inheritance) Baseline = 1.13; Passing Score >= 1.35
Grading
For each project, you will get assigned three tasks to test for that round: one each from easy, medium, and hard. For the first round, these will be randomly chosen. For subsequent rounds, we will carry two of your previous tasks forward. You can submit up to three versions (see deadlines on the calendar). For each task, we'll assign a score that captures the degree to which participants granted the correct permissions (what percent of the correct permissions do they have) and didn't grant the incorrect permissions (what percentage of the permissions that they shouldn't have that they don't) and efficiently (took the fastest path to get there). To get credit for a task, you need to outperform the baseline score by 20% for that task. Overall project grades will be:
- 93: 3 tasks that perform >20% over the baseline, spread across any iteration.
- 88: 2 tasks that perform >20% over the baseline, spread across any iteration.
- 83: 1 task that performs >20% over the baseline, in any iteration.
Then you can earn bonus points as follows:
- +7: all three tasks in a single iteration
- +3: two tasks in a single iteration
- +2: one of the top 3 performing tasks in the class (Note: you can earn this bonus for multiple tasks)
Prerequisite Readings
Baseline Code
Your repository contains the baseline version of the file permissions interface. Your assignment is to adapt that codebase to improve users' task performance using the ideas presented in Gender Mag. Within your top level repository directory, you should find a subfolder called gender-mag. Inside this folder, you will find the HTML files and JS files you'll need to edit. There are also wrappers HTML files that you can use to see the different scenarios.
To run the site locally, follow the directions in Running Sites Locally.
Two things to note:
- index.html loads the interface with no scenario, but since the scenario is tied to the interface, this isn't really a useful view. For development purposes, you may want to try starting with student_index.html which gives you a list of the scenarios to click on.
- By default, there is a survey that we'll ask mTurk workers to take that gets at their alignments with the different gender-mag facets. At development time, it's pretty annoying to have to click through it every time. To turn this off, you can go to view_baseline.html. Then, on the first line of the file set show_starter_dialogs = false
Introductory Studio
Testing Your Solutions
To determine whether your solution to a given scenario is correct you can:
- Complete the steps necessary to change the permissions to a state you believe is correct.
- Open the developer console in your browser.
- Type in validate_and_get_logs(). If your solution is correct, this will generate a long text blurb that will be what you need to include with your submission quiz. If your solution is not correct, you will get a list of erroneous permissions.
- You can also type in show_correct_permissions() to see what the correct set of permissions is.
Testing Your Site
To test your site, you need to do two things:
- Set up Github Pages
- Commit and push your code to Github. It may take a couple of minutes, but your changes will eventually be reflected in your Github pages site.
- Submit a sandbox hit that you can try out. (Submitting Sandbox HITs)
- If you open your developer's console in your browser and see red errors, then something is broken and will be broken when you submit your hits. You must fix it.
Setting Up Pages
To see what your version of the file permissions interface, you can view it through github pages. This will also be how mTurk users will view and interact with your site, so it's important to get going even if you plan to develop fully locally.
- From your github repository, go the the settings tab and scroll down to the bottom.
- Change the repository visibility to 'public.'
- Enable github Pages. Select the 'main' branch and the 'root' directory to serve pages from.
- Once you hit save, github will add a link to your hosted pages so that you can see them.
Uploading to Github Pages
To test or submit your project, you'll first need to upload your to Github.
- If you've made changes to javascript and/or typescript code, please rebuild your code to get the new changes. (see Node Setup - step 6)
- Test your site locally via localhost (see Running Sites Locally)
- Commit and push your changes (see Version Control in VS Code)
- It will take a few minutes but your Github pages site will eventually reflect your changes.
Making a Sandbox HIT
If you want to see what your MTurk workers will see, you can submit a sandbox hit.
- This page offers a walkthrough on how to do that. (Submitting Sandbox HITs)
Submission
For Iteration 0, you only need to upload your current site to AWS and submit the canvas quiz stating that you've done so.
For Iterations 1-3:
- Just as with testing, the first step in submitting your project is to upload your files to AWS.
- Keep in mind that your project will be tested by actual users on Mechanical Turk. If it breaks and they can't get paid, they may well rate you poorly as a requestor which can create challenges for you to get good data going forward. Please test your submission on AWS and via a sandbox hit before you declare it ready to go.
- After you've tested your submission, please make sure that you have $10 in your Mechanical Turk account.
- Last, please fill out the canvas submission form for the current iteration. This serves as your declaration that we can safely submit real HITs for you. We will randomly choose three of the tasks to test. For future iterations, we’ll keep two tasks the same and then choose a different third task. This is intended to keep disincentivize degenerate designs while also giving you an opportunity to iterate.
Note: when filling out the submission form, it will ask you for your log data. If you have not changed the basic flow of the interface (your changes add supplemental information but don't change what a user needs to do), you can answer "same" for all of these questions.
If you have made changes, then you'll need to do the following:
- For each changed scenario, navigate to the corresponding wrapper_*.html page in your version of the interface.
- Solve the scenario in as few clicks as possible in your interface.
- Once you think you have achieved the correct permissions required in the scenario, open the JavaScript console in your browser and type in validate_and_get_logs(). This will output one of two things:
If your solution is correct (i.e. it resulted in the right permissions to solve the scenario), a large text block will be printed out. These are the logs which give us the correct path for scoring MTurkers' attempts. This is the text you need to copy and paste into the submission question for each scenario.
If your solution is incorrect, in that the effective permissions state is not the one that solves this scenario, a series of errors and warnings will be printed out instead. The warnings list out what specifically is wrong with the current effective permission state: which permissions are allowed but should be denied according to the scenario, and vice versa. Try to understand why this is wrong and correct the problem.
Since accurate scoring depends on having the shortest/most efficent path, we highly recommend starting over and solving the scenario from scratch once you figure out how to get the correct answer. Otherwise your "shortest path" logs will include all of your trial-and-error.
Top 3 Solutions
add_new_user
- tiffanybalcarcel: https://cse-256-tiffanybalcarcel.s3.amazonaws.com/gender-mag/index.html
- coby.drexler: https://cse-256-coby.drexler.s3.amazonaws.com/gender-mag/index.html
- waits: https://cse-256-waits.s3.amazonaws.com/gender-mag/index.html
intern_permissions
- tiffanybalcarcel: https://cse-256-tiffanybalcarcel.s3.amazonaws.com/gender-mag/index.html
- coby.drexler: https://cse-256-coby.drexler.s3.amazonaws.com/gender-mag/index.html
- donghyunkim: https://cse-256-donghyunkim.s3.amazonaws.com/gender-mag/index.html
remove_direct_permission
- eqiu: https://cse-256-eqiu.s3.amazonaws.com/gender-mag/index.html
- sbishopp: https://cse-256-sbishopp.s3.amazonaws.com/gender-mag/index.html
- donghyunkim: https://cse-256-donghyunkim.s3.amazonaws.com/gender-mag/index.html
remove_inherited_permission
- coby.drexler: https://cse-256-coby.drexler.s3.amazonaws.com/gender-mag/index.html
- david.frankel: https://cse-256-david.frankel.s3.amazonaws.com/gender-mag/index.html
remove_user_with_inheritance
- sorayamoss: https://cse-256-sorayamoss.s3.amazonaws.com/gender-mag/index.html
- beyoglubora: https://cse-256-beyoglubora.s3.amazonaws.com/gender-mag/index.html
- sbishopp: https://cse-256-sbishopp.s3.amazonaws.com/gender-mag/index.html