Hacking the Browser Final Project: p5js Inspector(chrome extention)

For my final project, I made a chrome extension called p5js Inspector. p5js is a javascript library that people use to draw graphics and create interactive experience in the browser. This chrome extension allows people to manipulate shapes on their p5js canvas and visually understand the math that goes behind drawing stuff on canvas. Here's a video showing the use case, and here's my code on github.

Last semester I took a class called printing code where we use code to design and generate posters every week. 

I found that designers or people who are familiar with graphic user interface think about lines and locations differently from mathematicians and programmers. 

For example, designers are really good at using tools like Illustrator. We can view rulers, guide, grid system, and we can also manipulate shapes (move, resize, rotate them) on the canvas in real time. 

On the other hand, with many code-based drawing tools, like p5js, we usually start with estimating the coordinate system in our brain, and then guess our way out of the initial x,y position to draw something. After executing the code we see how close we get and we probably need to edit the code again. Drawing lines or ellipse might be easy, but drawing more complicate shapes like bezier curve& math might be overwhelming for beginners.

However, using computation to make art is much more powerful in terms of how complex and dense a digital piece of art can be made. So I feel like there should be a way to create a graphic user interface that allows us to code faster, better and also helps us understand the code in the process.

final.004.jpg

I don’t know the x,y,w,h of the shapes, so I wouldn’t be able to manipulate them. So in my p5.inspector.js, I changed p5 draw shapes functions a little. I saved all the info to a global array called myShapes, so I can get access to them in the DOM later. p5 creates window.rect. So I add sth to the function.

run_at, Controls when the files in js are injected. Can be "document_start", "document_end", or "document_idle". Defaults to "document_idle". 

In the case of "document_start", the files are injected before any other DOM is constructed or any other script is run. it’s too early for me, because in my p5.inspector.js, it needs to know p5 object and functions. 

In the case of "document_end", the files are injected immediately after the DOM is complete, but before subresources like images and frames have loaded. This is a good timing for me, because I after DOM is complete, p5 object and functions are in the DOM and are accessible for my script.

In the case of "document_idle", the browser chooses a time to inject scripts after the window.onload event fires. It’s too late for me because my script need to change p5 functions before window.onload. before p5 draw anything on the canvas.

I learned a lot about how the browser loads all the sources and build the DOM and when can we modify some functions and write it back before they get called.

Hacking the Browser: Chrome Extension

I made a chrome extension that can show the cursor’s x,y position in real time: https://github.com/yining1023/WYSIWYGdebugTool

Because p5 is not based on svg, I can’t select any shape from DOM, so I tried rune.js examples.

I made the second chrome extension to select the svg element on the canvas and show their watch points, e.g. line(x1, y1, x2, y2), watch points are x1, y1, x2, y2.

Todo list:

About SVG:

  • Show the watch points’ coordinates.
  • Show all the watch points for basic shapes (circle, ellipse, rectangle, triangle, beizer)
  • Making watching points draggable on the canvas
  • Showing the changing coordinates

Hacking the Browser - Bookmarklets

This week, I made a bookmarks that works against users. It will replace the cursor with a emoji image, and whenever users click the mouse, it will show more emoji images to block users sight. It's called "Raining Cats and Dogs"

Here's the vimeo video

For interesting bookmarklets, I didn't find an existing example, but I think it will be very cool to have a bookmarklet that hides your cursor when you click the mouse and show you the cursor when you click again. I think the way to implement it this:

var toggle = false;

document.getElementsByTagName('body')[0].onclick = function(){

if (toggle == false){

document.getElementsByTagName('body')[0].style = "cursor: none;"

toggle = true;

}

 

else{

document.getElementsByTagName('body')[0].style = "cursor: auto;"

toggle = false;

}

 

Hacking the Browser week1

For the first assignment, I made an interactive web app that allows people to shake their phones and decide what to eat today.

You can open this page on your phone: http://yining1023.github.io/whatToEat/

My code is here: https://github.com/yining1023/whatToEat

I used device orientation/motion API, shake.js, and emoji CSS.

You can see the video I posted on Instagram.

Here's some screenshots of my phone.

100 Days of Emojis and Interactive Web Design

TITLE:  100 Days of Emojis and Interactive Web Design

DATE:   January, 2016 - Present

TOOLKITS:   Javascript, P5.js, AngularJS

DESCRIPTION:
100 Days of Making is a new class at ITP that focuses on having a daily art practice for a period of 100 days.  I'm interested in emojis, drawing, design and web development, so I found a way to combine all my interests. I'm going to make one small interactive web design everyday under a theme of emojis. I hope I can get better at Javascript especially Angular as I iterate the design process over and over again. Everyday I will write down some notes here, and post a video about the web design on Tumblr and Instagram.

The code will be on my Github or Codepen.

My plan is to follow
- AngularJS tutorial

- Codepen

- Other good JS tutorial

Day21: drawing tool 2.0

http://jsfiddle.net/septends/yhz5Lno3/8/

Day20: drawing tool

http://jsfiddle.net/septends/yhz5Lno3/7/

Day19: paperjs pixel square

Day18: paperjs pixel circle

Day17: watermelon

Day16: change faces

Day15: Valentines day

angularJS: Scope, 

If we consider an AngularJS application to consist of:

  • View, which is the HTML.
  • Model, which is the data available for the current view.
  • Controller, which is the JavaScript function that makes/changes/removes/controls the data.

Then the scope is the Model.

The scope is a JavaScript object with properties and methods, which are available for both the view and the controller.

If you make changes in the view, the model and the controller will be updated.

Root Scope

All applications have a $rootScope which is the scope created on the HTML element that contains the ng-app directive.

The rootScope is available in the entire application.

If a variable has the same name in both the current scope and in the rootScope, the application use the one in the current scope.

AngularJS Filters

Filters can be added to expressions by using the pipe character |, followed by a filter.

Services: $location, $http, $timeout, $interval

Day14: WallPaper, angularJS

Day13: So Hungry, multi-particle system, Processing 

Day12: Moon #Processing

Day11: Too Much (work) #p5.js

Day10: Transition Glasses

Continue learning AngularJS, I learned ng-show in a hard way. I couldn't select element by id in AugularJS, I tried use angular.element, but it doesn't work. But angular has this ng-show makes things very easy.

<!-- when $scope.myValue is truthy (element is visible) -->
<div ng-show="myValue"></div>

<!-- when $scope.myValue is falsy (element is hidden) -->
<div ng-show="myValue" class="ng-hide"></div>

<img ng-show="name>=50" src="202.png">
<img ng-show="name<50" class="ng-hide" src="201.png">

chang the style of an element using: ng-style="{'opacity' : name/100}

Day9: Bookmarklet, replace cursor with emojis. see hacking the browser post.

Day8: Rock Paper Scissors

Day7: The width of the window

http://codepen.io/yining1023/pen/obPPjB

Day6: Emoji Toggle

code is on codepen: http://codepen.io/yining1023/debug/LGJGbL?editors=1100

Day5: emoji calculator

AngularJS Modules and Directives 

example: ng-binding

An AngularJS module defines an application.

The module is a container for the different parts of an application.

The module is a container for the application controllers.

Controllers always belong to a module.

Creating a Module:

angular.module

The "myApp" parameter refers to an HTML element in which the application will run.

Adding a Controller: 

ng-controller directive

Adding a Directive

AngularJS has a set of built-in directives which you can use to add functionality to your application.

***Functions can Pollute the Global Namespace

Global functions should be avoided in JavaScript. They can easily be overwritten or destroyed by other scripts.

AngularJS modules reduces this problem, by keeping all functions local to the module

When to Load the Library

it is recommended that you load the AngularJS library either in the <head> or at the start of the <body>.

This is because calls to angular.module can only be compiled after the library has been loaded.

ng-repeat

The ng-repeat directive actually clones HTML elements once for each item in a collection.

The ng-repeat directive used on an array of objects:

<li ng-repeat="x in names">
      {{ x }}
    </li>

*AngularJS is perfect for database CRUD (Create Read Update Delete) applications.
Just imagine if these objects were records from a database.

The ng-model directive binds the value of HTML controls (input, select, textarea) to application data.

Day4: See Hacking the Browser post

Day3: Intro to AngularJS and Expressions

Video: Tumblr and Instagram.

AngularJS is a JavaScript framework. It can be added to an HTML page with a <script> tag.

AngularJS extends HTML attributes with Directives, and binds data to HTML with Expressions.

  • The ng-app directive defines an AngularJS application.

  • The ng-model directive binds the value of HTML controls (input, select, textarea) to application data.
  • The ng-bind directive binds application data to the HTML view.
  • The ng-init directive initialize AngularJS application variables.
  • AngularJS expressions are written inside double braces: {{ expression }}.
  • AngularJS expressions bind AngularJS data to HTML the same way as the ng-bind directive.

<p>Name: <input type="text" ng-model="name"></p>
<p>{{name}}</p> // equals to <p ng-bind="name"></p>


  • AngularJS modules define AngularJS applications.
  • AngularJS controllers control AngularJS applications.

AngularJS Expressions

  • AngularJS will resolve the expression, and return the result exactly where the expression is written.
  • Unlike JavaScript expressions, AngularJS expressions can be written inside HTML.
  • AngularJS expressions do not support conditionals, loops, and exceptions, while JavaScript expressions do.
  • AngularJS expressions support filters, while JavaScript expressions do not.

Day2: Infinite Yawn

Video: Tumblr and Instagram.

 

Day 1: Leave Me Alone

Video: Tumblr and Instagram.

Screen Shot 2016-01-31 at 1.52.18 AM.png



Live Web

[Live web]Final Proposal

For live web final project, Tanya and I want to create an interactive story that people can participate in by using different controllers. So first, we will make a public web interface that lives on a big screen. Then we will ask people to use different controllers to move the story forward. We were inspired by human antenna circuit below. For now, when two people hold hands together to complete the circuit, the led will light up. We want to invite people to hold hands together and something interesting will happen on the web interface too. 

[video width="1280" height="720" m4v="http://www.yining1023.com/wp-content/uploads/2015/11/IMG_6091.m4v"][/video]

We explored other types of controller too.

  1. Hand LED Controller
  2. FSR
  3. Phone (shake/wavy motions)

We tried FSR control a rectangle in the webpage(the example from last class), that is fun too. We are also very interested in the idea of using phone as a controller. Because in this way, people who are at different geolocations can participate in the story too.

Here's system diagram 1.0:

          Web Interface(index.html) ← Server(server.js) →Client(client.js)

           → 1. Serial Communication → Sensor on Arduino(arduino sketch)

           → 2. Mobile Browser Interface (index.html)

Next step will be:

  1. Think of the stories we want to use
    1. fairytale?
    2. more feedback on that is
    3. ask people what do they think about holding hands together
  2. Making a web interface (screen)
  3. Get the controllers to work with the web interface

Live Web

[Live Web]Why is Slack successful?

For this week's assignment, I chose Slack. What amazed me about Slack is that it's based on a traditional interaction, messaging, and it looks like any other chatting tools, but it's a huge success today among many similar chatting tools. So I'm interested in finding out why Slack is so successful. Slack reminds me that even with a traditional interaction, if we put it into specific context, and add features around it, it still can be a huge success.

First of all, to me, Slack is a team collaboration tool. Its audiences are people working on projects or at offices. Here is an article from New York Times says, Slack is the office messaging app that may finally sink email. Here are some features of Slack.

  • Slack offers persistent chat rooms organized by topic, as well as private groups and direct messaging.
  • All content inside Slack is searchable, including files, conversations, and people.
  • Slack integrates with a large number of third-party services and supports community-built integrations. Major integrations include services such as Google Drive, Dropbox, Heroku, Crashlytics, GitHub, Runscope and Zendesk.
  • Slack also has voice, video, and screen sharing. Slack also added Emoji Reactions and user groups in 2015.

Slack works across every device you use. It provides mobile apps for iOS, Android, and Windows Phone (beta), in addition to their web browser client and native desktop clients for Mac OS X, Windows, and Linux (beta). Slack is also available for the Apple Watch, allowing users to send direct messages, see mentions, and make simple replies.

Slack has experienced rapid growth since it was first released. In February 2015, the company declared that it was adding around 10,000 new daily active users each week, and had more than 135,000 paying customers spread across 60,000 teams. By April, those numbers had grown to 200,000 paid subscribers and a total of 750,000 daily active users. In 2015, Slack passed more than a million daily active users.

Here are some of my ideas that why Slack is a huge success.

  1. From human-centered design perspective, instead of creating a whole new human habit, it's based a behavior that people are already used to, which is email. We already know how to use email, it's very easy for us to get familiar with Slack.
  2. It's perfectly suited for work, because it integrates with many third-party services, like GitHub, Google Drive and Heroku. It's especially convenient for programming team, for example, whenever someone updates code in GitHub, Slack will generate a post in the chatting group, and people can comment the post. I personally found this feature very useful.
  3. Slack has a very powerful searching engine. Everything in Slack—messages, notifications, files, and all—is automatically indexed and archived so that you can have it at your fingertips whenever you want. Slack also indexes the content of every file so you can search within PDFs, Word documents, Google docs, and more. This means that Slack will have a whole history of all your communications, and it's much clearer than email box.
  4. Slack represents a new working style at office. It encourage team work, collaboration, communication and more transparency at the office.

 

 

 

Live Web

[Live Web]Midterm proposal

My idea: I want to make something that allows people interact with HTML element through webcam. When people move their heads from left to right in front of the camera, they can control the movement of an avatar in the browser. When people move their hands, they can turn to the next page in the browser.

Moreover, I'd like to make a keyboard & mouse free space game. It's also a multiplayer game. Some players can be UFOs on the top side of the screen to shoot down spaceships. Other players can be spaceships on the bottom of the screen trying to survive. Here's my sketch.

IMG_5932

  1. Players allow browser to use their webcams
  2. Client program recognizes the head
  3. Client program tracks the movement of the head / hand
  4. Client moves the avatar around

For multiplayer games:

5. Client sends the current position data(Y axis data) to the server through socket

6. Server broadcasts all the positions to every client

7. Client gets the data, filters out it's own data, updates other player's position

Some research:

I found this Javascript library headtrackr for headtracking via webcam and WebRTC/getUserMedia very useful. But I have a feeling its tracking sensitivity might not be good enough for a game. But I will try my best. I also found this very cool JS library: Webcamwiper. It looks like the its very responsive.

I'm still not sure about tracking the movement of the head or the hand. I'm looking for some suggestions.

I'm also thinking to make the video itself.

Persuasion Design

[Persuasive Design]Week 5 Urban pain point, fun solutions

Pain point: So many trash bags laying on the street without recycling  New York City trash bags often pile up on the sidewalk, their contents strewn out onto the streets. The sheer volume of trash is enormous.

Of the residential waste, the city only recycles 15 percent, the New York Times reports. As for commercial waste, a recent report by Transform Don't Trash NYC says the city generates 5.5 million tons annually, out of which only about 25 percent is recycled.

Solution: 

Send people and stores securely-tied, heavy-duty opaque (non-transparent) bags. Each kind of bag is a kind of animal that represents one trash category.

Inspiration:

This bronze trash bag-inspired teddy bear is part of NYC Parks’ Art in the Parks program and is by Korean artist Gimhonsok. It aims to play on the ubiquitousness of garbage bags on the streets of New York City.

Artists in Manhattan are decorating the streets of NYC by gift wrapping garbage in polka dotted trash bags. The premise of the project is to beautify New York City and raise the public awareness of the arts and the environment.

508b0eb8e87fb397-DSC04157

the inflatables

Garbage Bag Dress

My sketch: 

IMG_5918

 

Persuasion Design

[Persuasive Design]Week 3 Design a Nudge

My pain point is getting up on time in the morning. I have a headache and feel very sleepy in the morning, and I am constantly a little late for classes that start in the morning. Clocky is a good example to wake people up in a playful way. I once wanted to design a coffee maker machine to make fresh coffee for me in the morning. But it's hard to implement this idea.

Then I realized the real reason that I couldn't get up in the morning is because I went to sleep too late. And for most of time, I was not working late at night, I was watching Netflix. Once I open Netflix, it will take me forever to close the tab. Imagine I use half of time which I spent on Netflix to do some work or go to sleep, my portfolio will be different and I won't have this pain point every morning.

I want to create something that won't force people to do anything. There are several exiting chrome extensions like Strict Workflow, which will open/close tab for users. I want to friendly remind me to close Netflix. People can make decision by themselves.

I created a chrome extension, called "Say Goodbye to Netflix".

0 1

Here's my code: https://github.com/yining1023/goodbyeNetflix

How to use it:

  1. Download code from here.
  2. Visit chrome://extensions in your browser .
  3. Ensure that the Developer mode checkbox in the top right-hand corner is checked.
  4. Click Load unpacked extension… to pop up a file-selection dialog.
  5. Navigate to the directory in which your extension files live, and select it.