- Lettuce构建过程
- 500天
- 500天与10000小时 @@ -507,7 +532,7 @@ $ git config --global user.email "电子邮箱"
- Knockout
- 当你写了一大堆测试,却不知道覆盖率有多少。
- 把光标移到int d中的d，按下shift+f6，输入result_sub
- 在弹出的窗口中输入mprint
- 选中Inline all invocations and remove the method(2 occurrences) 点确定
选中
@@ -1584,17 +1609,17 @@ React.render(对其进行
Extrace Method选择
basePrice再Inline Method- 最多的语言
- https://github.com/cujojs/when
- Knockout @@ -2802,7 +2827,7 @@ Set up your git name and email, this is important so that your commits can be id
- D3.js
- Dagre-D3.js @@ -2933,7 +2958,7 @@ Set up your git name and email, this is important so that your commits can be id
-GitHub可以托管各种git库，并提供一个web界面，但与其它像 SourceForge或Google Code这样的服务不同，GitHub的独特卖点在于从另外一个项目进行分支的简易性。为一个项目贡献代码非常简单：首先点击项目站点的“fork”的按钮，然后将代码检出并将修改加入到刚才分出的代码库中，最后通过内建的“pull request”机制向项目负责人申请代码合并。已经有人将GitHub称为代码玩家的MySpace。
在GitHub创建项目
+在 GitHub 创建项目
接着,我们试试在上面创建一个项目:
GitHub Roam @@ -532,15 +557,15 @@ git push -u origin mastergit remote add origin git@github.com:phodal/github-roam.git git push -u origin master如果你完成了上面的步骤之后,那么我想你想知道你需要怎样的项目。
-##GitHub流行项目分析
+GitHub 流行项目分析
之前曾经分析过一些GitHub的用户行为，现在我们先来说说GitHub上的Star吧。(截止: 2015年3月9日23时。)
-+
- - - - - + + + + + @@ -695,23 +720,23 @@ git push -u origin master 当只有一个人的时候，你只需要明确知道自己想要什么就够了。我们还需要的是CI、测试，以来提升代码的质量。
测试
通常我们都会找Document，如果没有的话，你会找什么?看源代码，还是看测试?
-
+it("specifying response when you need it", function (done) {-var doneFn = jasmine.createSpy("success");- -lettuce.get('/some/cool/url', function (result) {-expect(result).toEqual("awesome response");-done();-});- -expect(jasmine.Ajax.requests.mostRecent().url).toBe('/some/cool/url');-expect(doneFn).not.toHaveBeenCalled();- -jasmine.Ajax.requests.mostRecent().respondWith({-"status": 200,-"contentType": 'text/plain',-"responseText": 'awesome response'-});-});it("specifying response when you need it", function (done) { + var doneFn = jasmine.createSpy("success"); + + lettuce.get('/some/cool/url', function (result) { + expect(result).toEqual("awesome response"); + done(); + }); + + expect(jasmine.Ajax.requests.mostRecent().url).toBe('/some/cool/url'); + expect(doneFn).not.toHaveBeenCalled(); + + jasmine.Ajax.requests.mostRecent().respondWith({ + "status": 200, + "contentType": 'text/plain', + "responseText": 'awesome response' + }); +});代码来源: https://github.com/phodal/lettuce
上面的测试用例，清清楚楚地写明了用法，虽然写得有点扯。
等等，测试是用来干什么的。那么，先说说我为什么会想去写测试吧:
@@ -735,13 +760,13 @@ git push -u origin master ✓ should be able fadeout elements代码来源: https://github.com/phodal/lettuce
看上去似乎每个测试都很小，不过补完每一个测试之后我们就得到了测试覆盖率
-+
- - - - - + + + + + @@ -765,16 +790,16 @@ git push -u origin master 本地测试都通过了，于是我们添加了
Travis-CI来跑我们的测试CI
虽然node.js不算是一门语言，但是因为我们用的node，下面的是一个简单的
-.travis.yml示例:
+language: node_js-node_js:-- "0.10"- -notifications:-email: false- -before_install: npm install -g grunt-cli-install: npm install-after_success: CODECLIMATE_REPO_TOKEN=321480822fc37deb0de70a11931b4cb6a2a3cc411680e8f4569936ac8ffbb0ab codeclimate < coverage/lcov.infolanguage: node_js +node_js: + - "0.10" + +notifications: + email: false + +before_install: npm install -g grunt-cli +install: npm install +after_success: CODECLIMATE_REPO_TOKEN=321480822fc37deb0de70a11931b4cb6a2a3cc411680e8f4569936ac8ffbb0ab codeclimate < coverage/lcov.info代码来源: https://github.com/phodal/lettuce
我们把这些集成到
README.md之后，就有了之前那张图。CI对于一个开发者在不同城市开发同一项目上来说是很重要的，这意味着当你添加的部分功能有测试覆盖的时候，项目代码会更加强壮。
@@ -787,28 +812,28 @@ git push -u origin masterCode Climate是一个与github集成的工具，我们不仅仅可以看到测试覆盖率，还有代码质量。先看看上面的ajax类:
-
+Lettuce.get = function (url, callback) {-Lettuce.send(url, 'GET', callback);-};- -Lettuce.send = function (url, method, callback, data) {-data = data || null;-var request = new XMLHttpRequest();-if (callback instanceof Function) {-request.onreadystatechange = function () {-if (request.readyState === 4 && (request.status === 200 || request.status === 0)) {-callback(request.responseText);-}-};-}-request.open(method, url, true);-if (data instanceof Object) {-data = JSON.stringify(data);-request.setRequestHeader('Content-Type', 'application/json');-}-request.setRequestHeader('X-Requested-With', 'XMLHttpRequest');-request.send(data);-};Lettuce.get = function (url, callback) { + Lettuce.send(url, 'GET', callback); +}; + +Lettuce.send = function (url, method, callback, data) { + data = data || null; + var request = new XMLHttpRequest(); + if (callback instanceof Function) { + request.onreadystatechange = function () { + if (request.readyState === 4 && (request.status === 200 || request.status === 0)) { + callback(request.responseText); + } + }; + } + request.open(method, url, true); + if (data instanceof Object) { + data = JSON.stringify(data); + request.setRequestHeader('Content-Type', 'application/json'); + } + request.setRequestHeader('X-Requested-With', 'XMLHttpRequest'); + request.send(data); +};代码来源: https://github.com/phodal/lettuce
在Code Climate在出现了一堆问题
-
@@ -855,24 +880,24 @@ git push -u origin master
好在Knockout可以用Require.js进行管理，于是，使用了
-Require.js进行管理:
+<script type="text/javascript" data-main="app/scripts/main.js" src="app/lib/require.js"></script>
-main.js配置如下:
+require.config({-baseUrl: 'app',-paths:{-jquery: 'lib/jquery',-json: 'lib/json',-text: 'lib/text'-}-});- -require(['scripts/ko-bindings']);- -require(['lib/knockout', 'scripts/TalentTree', 'json!data/web.json'], function(ko, TalentTree, TalentData) {-'use strict';-var vm = new TalentTree(TalentData);-ko.applyBindings(vm);-});require.config({ + baseUrl: 'app', + paths:{ + jquery: 'lib/jquery', + json: 'lib/json', + text: 'lib/text' + } +}); + +require(['scripts/ko-bindings']); + +require(['lib/knockout', 'scripts/TalentTree', 'json!data/web.json'], function(ko, TalentTree, TalentData) { + 'use strict'; + var vm = new TalentTree(TalentData); + ko.applyBindings(vm); +});text、json插件主要是用于处理web.json，即用json来处理技能，于是不同的类到了不同的js文件。
|____TalentTree.js |____Utils.js. |____Book.js @@ -884,52 +909,52 @@ git push -u origin master加上了后来的推荐阅读书籍等等。而Book和Link都是继承自Doc。
-
+define(['scripts/Doc'], function(Doc) {-'use strict';-function Book(_e) {-Doc.apply(this, arguments);-}-Book.prototype = new Doc();- -return Book;-});define(['scripts/Doc'], function(Doc) { + 'use strict'; + function Book(_e) { + Doc.apply(this, arguments); + } + Book.prototype = new Doc(); + + return Book; +});而这里便是后面对其进行重构的内容。Doc类则是Skillock中类的一个缩影
-
+define([], function() {-'use strict';-var Doc = function (_e) {-var e = _e || {};-var self = this;- -self.label = e.label || (e.url || 'Learn more');-self.url = e.url || 'javascript:void(0)';-};- -return Doc;-});define([], function() { + 'use strict'; + var Doc = function (_e) { + var e = _e || {}; + var self = this; + + self.label = e.label || (e.url || 'Learn more'); + self.url = e.url || 'javascript:void(0)'; + }; + + return Doc; +});或者说这是一个AMD的Class应该有的样子。考虑到this的隐性绑定，作者用了self=this来避免这个问题。最后Return了这个对象，我们在调用的就需要new一个。大部分在代码中返回的都是对象，除了在Utils类里面返回的是函数:
-
+return {-getSkillsByHash: getSkillsByHash,-getSkillById: getSkillById,-prettyJoin: prettyJoin-};return { + getSkillsByHash: getSkillsByHash, + getSkillById: getSkillById, + prettyJoin: prettyJoin +};当然函数也是一个对象。
自动化测试
一直习惯用Travis CI，于是也继续用Travis Ci，
-.travis.yml配置如下所示:
+language: node_js-node_js:-- "0.10"- -notifications:-email: false- -branches:-only:-- gh-pageslanguage: node_js +node_js: + - "0.10" + +notifications: + email: false + +branches: + only: + - gh-pages使用gh-pages的原因是，我们一push代码的时候，就可以自动测试、部署等等，好处一堆堆的。
接着我们需要在
-package.json里面添加脚本
+"scripts": {-"test": "mocha"-}这样当我们push代码的时候便会自动跑所有的测试。因为mocha的主要配置是用
mocha.opts，所以我们还需要配置一下mocha.opts--reporter spec --ui bdd @@ -945,7 +970,7 @@ test/specline 5 col 25 A constructor name should start with an uppercase letter. line 21 col 62 Strings must use singlequote.这是一种驱动写出更规范js的方法。
-###Mocha
+Mocha
@@ -957,55 +982,55 @@ line 21 col 62 Strings must use singlequote. ✓ should return link label & urlMocha 是一个优秀的JS测试框架，支持TDD/BDD，结合 should.js/expect/chai/better-assert，能轻松构建各种风格的测试用例。
测试示例
简单地看一下Book的测试:
-
+/* global describe, it */- -var requirejs = require("requirejs");-var assert = require("assert");-var should = require("should");-requirejs.config({-baseUrl: 'app/',-nodeRequire: require-});- -describe('Book,Link', function () {-var Book, Link;-before(function (done) {-requirejs(['scripts/Book'、], function (Book_Class) {-Book = Book_Class;-done();-});-});- -describe('Book Test', function () {-it('should return book label & url', function () {-var book_name = 'Head First HTML与CSS';-var url = 'http://www.phodal.com';-var books = {-label: book_name,-url: url-};- -var _book = new Book(books);-_book.label.should.equal(book_name);-_book.url.should.equal(url);-});-});-});/* global describe, it */ + +var requirejs = require("requirejs"); +var assert = require("assert"); +var should = require("should"); +requirejs.config({ + baseUrl: 'app/', + nodeRequire: require +}); + +describe('Book,Link', function () { + var Book, Link; + before(function (done) { + requirejs(['scripts/Book'、], function (Book_Class) { + Book = Book_Class; + done(); + }); + }); + + describe('Book Test', function () { + it('should return book label & url', function () { + var book_name = 'Head First HTML与CSS'; + var url = 'http://www.phodal.com'; + var books = { + label: book_name, + url: url + }; + + var _book = new Book(books); + _book.label.should.equal(book_name); + _book.url.should.equal(url); + }); + }); +});因为我们用
-require.js来管理浏览器端，在后台写测试来测试的时候，我们也需要用他来管理我们的依赖，这也就是为什么这个测试这么长的原因，多数情况下一个测试类似于这样子的。(用Jasmine似乎会是一个更好的主意，但是用习惯Jasmine了)
+describe('Book Test', function () {-it('should return book label & url', function () {-var book_name = 'Head First HTML与CSS';-var url = 'http://www.phodal.com';-var books = {-label: book_name,-url: url-};- -var _book = new Book(books);-_book.label.should.equal(book_name);-_book.url.should.equal(url);-});-});describe('Book Test', function () { +it('should return book label & url', function () { + var book_name = 'Head First HTML与CSS'; + var url = 'http://www.phodal.com'; + var books = { + label: book_name, + url: url + }; + + var _book = new Book(books); + _book.label.should.equal(book_name); + _book.url.should.equal(url); +}); +});最后的断言，也算是测试的核心，保证测试是有用的。
代码质量与重构
-
@@ -1013,7 +1038,7 @@ line 21 col 62 Strings must use singlequote.
这就是个问题了，于是偶然间看到了一个叫code climate的网站。
-###Code Climate
+Code Climate
@@ -1131,89 +1156,89 @@ line 21 col 62 Strings must use singlequote.Code Climate consolidates the results from a suite of static analysis tools into a single, real-time report, giving your team the information it needs to identify hotspots, evaluate new approaches, and improve code quality.
分享得到的最后的结果是:
[Coverage][1]
-###代码的坏味道
+代码的坏味道
于是我们就打开
lib/database/sqlite_helper.js，因为其中有两个坏味道Similar code found in two :expression_statement nodes (mass = 86)
在代码的
-lib/database/sqlite_helper.js:58…61 < >
+SQLiteHelper.prototype.deleteData = function (url, callback) {-'use strict';-var sql_command = "DELETE FROM " + config.table_name + " where " + URLHandler.getKeyFromURL(url) + "=" + URLHandler.getValueFromURL(url);-SQLiteHelper.prototype.basic(sql_command, callback);SQLiteHelper.prototype.deleteData = function (url, callback) { + 'use strict'; + var sql_command = "DELETE FROM " + config.table_name + " where " + URLHandler.getKeyFromURL(url) + "=" + URLHandler.getValueFromURL(url); + SQLiteHelper.prototype.basic(sql_command, callback);lib/database/sqlite_helper.js:64…67 < >
与
-
+SQLiteHelper.prototype.getData = function (url, callback) {-'use strict';-var sql_command = "SELECT * FROM " + config.table_name + " where " + URLHandler.getKeyFromURL(url) + "=" + URLHandler.getValueFromURL(url);-SQLiteHelper.prototype.basic(sql_command, callback);SQLiteHelper.prototype.getData = function (url, callback) { + 'use strict'; + var sql_command = "SELECT * FROM " + config.table_name + " where " + URLHandler.getKeyFromURL(url) + "=" + URLHandler.getValueFromURL(url); + SQLiteHelper.prototype.basic(sql_command, callback);只是这是之前修改过的重复。。
原来的代码是这样的
-
+SQLiteHelper.prototype.postData = function (block, callback) {-'use strict';-var db = new sqlite3.Database(config.db_name);-var str = this.parseData(config.keys);-var string = this.parseData(block);- -var sql_command = "insert or replace into " + config.table_name + " (" + str + ") VALUES (" + string + ");";-db.all(sql_command, function (err) {-SQLiteHelper.prototype.errorHandler(err);-db.close();-callback();-});-};- -SQLiteHelper.prototype.deleteData = function (url, callback) {-'use strict';-var db = new sqlite3.Database(config.db_name);-var sql_command = "DELETE FROM " + config.table_name + " where " + URLHandler.getKeyFromURL(url) + "=" + URLHandler.getValueFromURL(url);-db.all(sql_command, function (err) {-SQLiteHelper.prototype.errorHandler(err);-db.close();-callback();-});-};- -SQLiteHelper.prototype.getData = function (url, callback) {-'use strict';-var db = new sqlite3.Database(config.db_name);-var sql_command = "SELECT * FROM " + config.table_name + " where " + URLHandler.getKeyFromURL(url) + "=" + URLHandler.getValueFromURL(url);-db.all(sql_command, function (err, rows) {-SQLiteHelper.prototype.errorHandler(err);-db.close();-callback(JSON.stringify(rows));-});-};SQLiteHelper.prototype.postData = function (block, callback) { + 'use strict'; + var db = new sqlite3.Database(config.db_name); + var str = this.parseData(config.keys); + var string = this.parseData(block); + + var sql_command = "insert or replace into " + config.table_name + " (" + str + ") VALUES (" + string + ");"; + db.all(sql_command, function (err) { + SQLiteHelper.prototype.errorHandler(err); + db.close(); + callback(); + }); +}; + +SQLiteHelper.prototype.deleteData = function (url, callback) { + 'use strict'; + var db = new sqlite3.Database(config.db_name); + var sql_command = "DELETE FROM " + config.table_name + " where " + URLHandler.getKeyFromURL(url) + "=" + URLHandler.getValueFromURL(url); + db.all(sql_command, function (err) { + SQLiteHelper.prototype.errorHandler(err); + db.close(); + callback(); + }); +}; + +SQLiteHelper.prototype.getData = function (url, callback) { + 'use strict'; + var db = new sqlite3.Database(config.db_name); + var sql_command = "SELECT * FROM " + config.table_name + " where " + URLHandler.getKeyFromURL(url) + "=" + URLHandler.getValueFromURL(url); + db.all(sql_command, function (err, rows) { + SQLiteHelper.prototype.errorHandler(err); + db.close(); + callback(JSON.stringify(rows)); + }); +};说的也是大量的重复，重构完的代码
-
+SQLiteHelper.prototype.basic = function(sql, db_callback){-'use strict';-var db = new sqlite3.Database(config.db_name);-db.all(sql, function (err, rows) {-SQLiteHelper.prototype.errorHandler(err);-db.close();-db_callback(JSON.stringify(rows));-});- -};- -SQLiteHelper.prototype.postData = function (block, callback) {-'use strict';-var str = this.parseData(config.keys);-var string = this.parseData(block);- -var sql_command = "insert or replace into " + config.table_name + " (" + str + ") VALUES (" + string + ");";-SQLiteHelper.prototype.basic(sql_command, callback);-};- -SQLiteHelper.prototype.deleteData = function (url, callback) {-'use strict';-var sql_command = "DELETE FROM " + config.table_name + " where " + URLHandler.getKeyFromURL(url) + "=" + URLHandler.getValueFromURL(url);-SQLiteHelper.prototype.basic(sql_command, callback);-};- -SQLiteHelper.prototype.getData = function (url, callback) {-'use strict';-var sql_command = "SELECT * FROM " + config.table_name + " where " + URLHandler.getKeyFromURL(url) + "=" + URLHandler.getValueFromURL(url);-SQLiteHelper.prototype.basic(sql_command, callback);-};SQLiteHelper.prototype.basic = function(sql, db_callback){ + 'use strict'; + var db = new sqlite3.Database(config.db_name); + db.all(sql, function (err, rows) { + SQLiteHelper.prototype.errorHandler(err); + db.close(); + db_callback(JSON.stringify(rows)); + }); + +}; + +SQLiteHelper.prototype.postData = function (block, callback) { + 'use strict'; + var str = this.parseData(config.keys); + var string = this.parseData(block); + + var sql_command = "insert or replace into " + config.table_name + " (" + str + ") VALUES (" + string + ");"; + SQLiteHelper.prototype.basic(sql_command, callback); +}; + +SQLiteHelper.prototype.deleteData = function (url, callback) { + 'use strict'; + var sql_command = "DELETE FROM " + config.table_name + " where " + URLHandler.getKeyFromURL(url) + "=" + URLHandler.getValueFromURL(url); + SQLiteHelper.prototype.basic(sql_command, callback); +}; + +SQLiteHelper.prototype.getData = function (url, callback) { + 'use strict'; + var sql_command = "SELECT * FROM " + config.table_name + " where " + URLHandler.getKeyFromURL(url) + "=" + URLHandler.getValueFromURL(url); + SQLiteHelper.prototype.basic(sql_command, callback); +};重构完后的代码比原来还长，这似乎是个问题~~
Git 提交信息及几种不同的规范
@@ -1354,81 +1379,81 @@ React.render(
代码及setup请见github: js-refactor
代码说明
-uMarkdown是一个用于将Markdown转化为HTML的库。代码看上去就像一个很典型的过程代码:
+/* code */-while ((stra = micromarkdown.regexobject.code.exec(str)) !== null) {-str = str.replace(stra[0], '<code>\n' + micromarkdown.htmlEncode(stra[1]).replace(/\n/gm, '<br/>').replace(/\ /gm, ' ') + '</code>\n');-}- -/* headlines */-while ((stra = micromarkdown.regexobject.headline.exec(str)) !== null) {-count = stra[1].length;-str = str.replace(stra[0], '<h' + count + '>' + stra[2] + '</h' + count + '>' + '\n');-}- -/* mail */-while ((stra = micromarkdown.regexobject.mail.exec(str)) !== null) {-str = str.replace(stra[0], '<a href="mailto:' + stra[1] + '">' + stra[1] + '</a>');-}/* code */ +while ((stra = micromarkdown.regexobject.code.exec(str)) !== null) { + str = str.replace(stra[0], '<code>\n' + micromarkdown.htmlEncode(stra[1]).replace(/\n/gm, '<br/>').replace(/\ /gm, ' ') + '</code>\n'); +} + +/* headlines */ +while ((stra = micromarkdown.regexobject.headline.exec(str)) !== null) { + count = stra[1].length; + str = str.replace(stra[0], '<h' + count + '>' + stra[2] + '</h' + count + '>' + '\n'); +} + +/* mail */ +while ((stra = micromarkdown.regexobject.mail.exec(str)) !== null) { + str = str.replace(stra[0], '<a href="mailto:' + stra[1] + '">' + stra[1] + '</a>'); +}选这个做重构的开始，不仅仅是因为之前在写EchoesWorks的时候进行了很多的重构。而且它更适合于
重构到设计模式的理论。让我们在重构完之后，给作者进行pull request吧。Markdown的解析过程，有点类似于
Pipe and Filters模式(架构模式)。Filter即我们在代码中看到的正规表达式集:
-
+regexobject: {-headline: /^(\#{1,6})([^\#\n]+)$/m,-code: /\s\`\`\`\n?([^`]+)\`\`\`/g他会匹配对应的Markdown类型，随后进行替换和处理。而``str```，就是管理口的输入和输出。
接着，我们就可以对其进行简单的重构。
(ps: 推荐用WebStrom来做重构，自带重构功能)
作为一个示例，我们先提出codeHandler方法，即将上面的
-
+/* code */-while ((stra = micromarkdown.regexobject.code.exec(str)) !== null) {-str = str.replace(stra[0], '<code>\n' + micromarkdown.htmlEncode(stra[1]).replace(/\n/gm, '<br/>').replace(/\ /gm, ' ') + '</code>\n');-}/* code */ +while ((stra = micromarkdown.regexobject.code.exec(str)) !== null) { + str = str.replace(stra[0], '<code>\n' + micromarkdown.htmlEncode(stra[1]).replace(/\n/gm, '<br/>').replace(/\ /gm, ' ') + '</code>\n'); +}提取方法成
-
+codeFilter: function (str, stra) {-return str.replace(stra[0], '<code>\n' + micromarkdown.htmlEncode(stra[1]).replace(/\n/gm, '<br/>').replace(/\ /gm, ' ') + '</code>\n');-},codeFilter: function (str, stra) { + return str.replace(stra[0], '<code>\n' + micromarkdown.htmlEncode(stra[1]).replace(/\n/gm, '<br/>').replace(/\ /gm, ' ') + '</code>\n'); + },while语句就成了
-
+while ((stra = regexobject.code.exec(str)) !== null) {-str = this.codeFilter(str, stra);-}然后，运行所有的测试。
grunt test同理我们就可以
-mail、headline等方法进行重构。接着就会变成类似于下面的代码，
+/* code */-while ((execStr = regExpObject.code.exec(str)) !== null) {-str = codeHandler(str, execStr);-}- -/* headlines */-while ((execStr = regExpObject.headline.exec(str)) !== null) {-str = headlineHandler(str, execStr);-}- -/* lists */-while ((execStr = regExpObject.lists.exec(str)) !== null) {-str = listHandler(str, execStr);-}- -/* tables */-while ((execStr = regExpObject.tables.exec(str)) !== null) {-str = tableHandler(str, execStr, strict);-}/* code */ +while ((execStr = regExpObject.code.exec(str)) !== null) { +str = codeHandler(str, execStr); +} + +/* headlines */ +while ((execStr = regExpObject.headline.exec(str)) !== null) { +str = headlineHandler(str, execStr); +} + +/* lists */ +while ((execStr = regExpObject.lists.exec(str)) !== null) { +str = listHandler(str, execStr); +} + +/* tables */ +while ((execStr = regExpObject.tables.exec(str)) !== null) { +str = tableHandler(str, execStr, strict); +}然后你也看到了，上面有一堆重复的代码，接着让我们用JavaScript的
-奇技淫巧，即apply方法，把上面的重复代码变成。
+['code', 'headline', 'lists', 'tables', 'links', 'mail', 'url', 'smlinks', 'hr'].forEach(function (type) {-while ((stra = regexobject[type].exec(str)) !== null) {-str = that[(type + 'Handler')].apply(that, [stra, str, strict]);-}-});['code', 'headline', 'lists', 'tables', 'links', 'mail', 'url', 'smlinks', 'hr'].forEach(function (type) { + while ((stra = regexobject[type].exec(str)) !== null) { + str = that[(type + 'Handler')].apply(that, [stra, str, strict]); + } +});进行测试，blabla，都是过的。
-
+Markdown-✓ should parse h1~h3-✓ should parse link-✓ should special link-✓ should parse font style-✓ should parse code-✓ should parse ul list-✓ should parse ul table-✓ should return correctly class nameMarkdown + ✓ should parse h1~h3 + ✓ should parse link + ✓ should special link + ✓ should parse font style + ✓ should parse code + ✓ should parse ul list + ✓ should parse ul table + ✓ should return correctly class name快来试试吧， https://github.com/artisanstack/js-refactor
是时候讨论这个Refactor利器了，最初看到这个重构的过程是从ThoughtWorks郑大晔校开始的，只是之前对于Java的另外一个编辑器Eclipse的坏感。。这些在目前已经不是很重要了，试试这个公司里面应用广泛的编辑器。
Intellij Idea重构
@@ -1436,18 +1461,18 @@ React.render(编写测试->功能代码->修改测试->重构上次在和buddy聊天的时候，才知道测试在功能简单的时候是后行的，在功能复杂不知道怎么下手的时候是先行的。
开始之前请原谅我对于Java语言的一些无知，然后，看一下我写的Main函数：
-
+package com.phodal.learing;- -public class Main {- -public static void main(String[] args) {-int c=new Cal().add(1,2);-int d=new Cal2().sub(2,1);-System.out.println("Hello,s");-System.out.println(c);-System.out.println(d);-}-}package com.phodal.learing; + +public class Main { + + public static void main(String[] args) { + int c=new Cal().add(1,2); + int d=new Cal2().sub(2,1); + System.out.println("Hello,s"); + System.out.println(c); + System.out.println(d); + } +}代码写得还好(自我感觉)，先不管Cal和Cal2两个类。大部分都能看懂，除了c,d不知道他们表达的是什么意思，于是。
Rename
快捷键:Shift+F6
@@ -1457,18 +1482,18 @@ React.render(于是就有
-
+package com.phodal.learing;- -public class Main {- -public static void main(String[] args) {-int result_add=new Cal().add(1,2);-int result_sub=new Cal2().sub(2,1);-System.out.println("Hello,s");-System.out.println(result_add);-System.out.println(result_sub);-}-}package com.phodal.learing; + +public class Main { + + public static void main(String[] args) { + int result_add=new Cal().add(1,2); + int result_sub=new Cal2().sub(2,1); + System.out.println("Hello,s"); + System.out.println(result_add); + System.out.println(result_sub); + } +}Extract Method
快捷键:alt+command+m
作用:扩展方法
@@ -1478,17 +1503,17 @@ React.render(于是有了
-
+public static void main(String[] args) {-int result_add=new Cal().add(1,2);-int result_sub=new Cal2().sub(2,1);-System.out.println("Hello,s");-mprint(result_add);-mprint(result_sub);-}- -private static void mprint(int result_sub) {-System.out.println(result_sub);-}public static void main(String[] args) { + int result_add=new Cal().add(1,2); + int result_sub=new Cal2().sub(2,1); + System.out.println("Hello,s"); + mprint(result_add); + mprint(result_sub); +} + +private static void mprint(int result_sub) { + System.out.println(result_sub); +}似乎我们不应该这样对待System.out.println，那么让我们内联回去
Inline Method
快捷键:alt+command+n
@@ -1499,41 +1524,41 @@ React.render(然后我们等于什么也没有做了~~:
-
+public static void main(String[] args) {-int result_add=new Cal().add(1,2);-int result_sub=new Cal2().sub(2,1);-System.out.println("Hello,s");-System.out.println(result_add);-System.out.println(result_sub);-}public static void main(String[] args) { + int result_add=new Cal().add(1,2); + int result_sub=new Cal2().sub(2,1); + System.out.println("Hello,s"); + System.out.println(result_add); + System.out.println(result_sub); +}似乎这个例子不是很好，但是够用来说明了。
Pull Members Up
开始之前让我们先看看Cal2类:
-
+public class Cal2 extends Cal {- -public int sub(int a,int b){-return a-b;-}-}以及Cal2的父类Cal
-
+public class Cal {- -public int add(int a,int b){-return a+b;-}- -}public class Cal { + + public int add(int a,int b){ + return a+b; + } + +}最后的结果，就是将Cal2类中的sub方法，提到父类:
-
+public class Cal {- -public int add(int a,int b){-return a+b;-}- -public int sub(int a,int b){-return a-b;-}-}public class Cal { + + public int add(int a,int b){ + return a+b; + } + + public int sub(int a,int b){ + return a-b; + } +}而我们所要做的就是鼠标右键
重构之以查询取代临时变量
快捷键
@@ -1541,42 +1566,42 @@ React.render(Windows/Linux: 木有
或者:
Shift+alt+command+T再选择Replace Temp with Query鼠标: Refactor |
-Replace Temp with Query####重构之前
+重构之前
过多的临时变量会让我们写出更长的函数，函数不应该太多，以便使功能单一。这也是重构的另外的目的所在，只有函数专注于其功能，才会更容易读懂。
以书中的代码为例
-
+import java.lang.System;- -public class replaceTemp {-public void count() {-double basePrice = _quantity * _itemPrice;-if (basePrice > 1000) {-return basePrice * 0.95;-} else {-return basePrice * 0.98;-}-}-}import java.lang.System; + +public class replaceTemp { + public void count() { + double basePrice = _quantity * _itemPrice; + if (basePrice > 1000) { + return basePrice * 0.95; + } else { + return basePrice * 0.98; + } + } +}重构
选中
basePrice很愉快地拿鼠标点上面的重构
Replace Temp With Query 便会返回
-
+import java.lang.System;- -public class replaceTemp {-public void count() {-if (basePrice() > 1000) {-return basePrice() * 0.95;-} else {-return basePrice() * 0.98;-}-}- -private double basePrice() {-return _quantity * _itemPrice;-}-}import java.lang.System; + +public class replaceTemp { + public void count() { + if (basePrice() > 1000) { + return basePrice() * 0.95; + } else { + return basePrice() * 0.98; + } + } + + private double basePrice() { + return _quantity * _itemPrice; + } +}而实际上我们也可以
####Intellij IDEA重构
+Intellij IDEA重构
在Intellij IDEA的文档中对此是这样的例子
-
+public class replaceTemp {- -public void method() {-String str = "str";-String aString = returnString().concat(str);-System.out.println(aString);-}- -}public class replaceTemp { + + public void method() { + String str = "str"; + String aString = returnString().concat(str); + System.out.println(aString); + } + +}接着我们选中
aString，再打开重构菜单，或者Command+Alt+Shift+T再选中Replace Temp with Query便会有下面的结果:
@@ -1620,28 +1645,28 @@ public class replaceTemp {之前正在重写一个物联网的服务端，主要便是结合CoAP、MQTT、HTTP等协议构成一个物联网的云服务。现在，主要的任务是集中于协议与授权。由于，不同协议间的授权是不一样的，最开始的时候我先写了一个http put授权的功能，而在起先的时候是如何测试的呢?
curl --user root:root -X PUT -d '{ "dream": 1 }' -H "Content-Type: application/json" http://localhost:8899/topics/test我只要顺利在request中看有无
-req.headers.authorization，我便可以继续往下，接着给个判断。毕竟，我们对HTTP协议还是蛮清楚的。
+if (!req.headers.authorization) {-res.statusCode = 401;-res.setHeader('WWW-Authenticate', 'Basic realm="Secure Area"');-return res.end('Unauthorized');-}if (!req.headers.authorization) { + res.statusCode = 401; + res.setHeader('WWW-Authenticate', 'Basic realm="Secure Area"'); + return res.end('Unauthorized'); +}可是除了HTTP协议，还有MQTT和CoAP。对于MQTT协议来说，那还算好，毕竟自带授权，如:
-
+mosquitto_pub -u root -P root -h localhost -d -t lettuce -m "Hello, MQTT. This is my first message."mosquitto_pub -u root -P root -h localhost -d -t lettuce -m "Hello, MQTT. This is my first message."便可以让我们简单地完成这个功能，然而有的协议是没有这样的功能如CoAP协议中是用Option来进行授权的。现在的工具如libcoap只能有如下的简单功能
-
+coap-client -m get coap://127.0.0.1:5683/topics/zero -T于是，先写了个测试脚本来验证功能。
-
+var coap = require('coap');-var request = coap.request;-var req = request({hostname: 'localhost',port:5683,pathname: '',method: 'POST'});- -...- -req.setHeader("Accept", "application/json");-req.setOption('Block2', [new Buffer('phodal'), new Buffer('phodal')]);- -...- -req.end();var coap = require('coap'); +var request = coap.request; +var req = request({hostname: 'localhost',port:5683,pathname: '',method: 'POST'}); + +... + +req.setHeader("Accept", "application/json"); +req.setOption('Block2', [new Buffer('phodal'), new Buffer('phodal')]); + +... + +req.end();写完测试脚本后发现不对了，这个不应该是测试的代码吗? 于是将其放到了spec中，接着发现了上面的全部功能的实现过程为什么不用TDD实现呢？
说说TDD
测试驱动开发是一个很“古老”的程序开发方法，然而由于国内的开发流程的问题——即开发人员负责功能的测试，导致这么好的一项技术没有在国内推广。
@@ -1672,7 +1697,7 @@ public class replaceTemp {看了一下源码，大概原理就是用
requests下载html，接着用lxml解析html，比较有意思的是内嵌了一个DSL。这是一个Python的库。
-pip install twill###Twill 登陆测试
+Twill 登陆测试
1.启动我们的应用。
2.进入twill shell
Note: submit is using submit button: name="login", value="登入" current page: http://127.0.0.1:5000/twill-sh @@ -1704,7 +1729,7 @@ fv 1 password test发现重定向到首页了。
-###Twill 测试脚本
+Twill 测试脚本
当然我们也可以用脚本直接来测试
login.twill:go http://127.0.0.1:5000/login @@ -2030,37 +2055,37 @@ Set up your git name and email, this is important so that your commits can be id==, 这个文件代表什么？
2014年1月1日零时到一时，用户在github上的操作，这里的用户指的是很多。。一共有4814条数据，从commit、create到issues都有。
数据解析
-
+import json-for line in open(jsonfile):-line = f.readline()然后再解析json
-
+import dateutil.parser- -lin = json.loads(line)-date = dateutil.parser.parse(lin["created_at"])这里用到了
-dateutil，因为新鲜出炉的数据是string需要转换为dateutil，再到数据放到数组里头。最后有就有了parse_data
+def parse_data(jsonfile):-f = open(jsonfile, "r")-dataarray = []-datacount = 0- -for line in open(jsonfile):-line = f.readline()-lin = json.loads(line)-date = dateutil.parser.parse(lin["created_at"])-datacount += 1-dataarray.append(date.minute)- -minuteswithcount = [(x, dataarray.count(x)) for x in set(dataarray)]-f.close()-return minuteswithcountdef parse_data(jsonfile): + f = open(jsonfile, "r") + dataarray = [] + datacount = 0 + + for line in open(jsonfile): + line = f.readline() + lin = json.loads(line) + date = dateutil.parser.parse(lin["created_at"]) + datacount += 1 + dataarray.append(date.minute) + + minuteswithcount = [(x, dataarray.count(x)) for x in set(dataarray)] + f.close() + return minuteswithcount下面这句代码就是将上面的解析为
-
+minuteswithcount = [(x, dataarray.count(x)) for x in set(dataarray)]这样的数组以便于解析
-
+[(0, 92), (1, 67), (2, 86), (3, 73), (4, 76), (5, 67), (6, 61), (7, 71), (8, 62), (9, 71), (10, 70), (11, 79), (12, 62), (13, 67), (14, 76), (15, 67), (16, 74), (17, 48), (18, 78), (19, 73), (20, 89), (21, 62), (22, 74), (23, 61), (24, 71), (25, 49), (26, 59), (27, 59), (28, 58), (29, 74), (30, 69), (31, 59), (32, 89), (33, 67), (34, 66), (35, 77), (36, 64), (37, 71), (38, 75), (39, 66), (40, 62), (41, 77), (42, 82), (43, 95), (44, 77), (45, 65), (46, 59), (47, 60), (48, 54), (49, 66), (50, 74), (51, 61), (52, 71), (53, 90), (54, 64), (55, 67), (56, 67), (57, 55), (58, 68), (59, 91)]Matplotlib
开始之前需要安装``matplotlib
-
+sudo pip install matplotlib然后引入这个库
import matplotlib.pyplot as plt如上面的那个结果，只需要
@@ -2071,47 +2096,47 @@ Set up your git name and email, this is important so that your commits can be id plt.show()最后代码可见
-
+#!/usr/bin/env python-# -*- coding: utf-8 -*-- -import json-import dateutil.parser-import numpy as np-import matplotlib.mlab as mlab-import matplotlib.pyplot as plt- - -def parse_data(jsonfile):-f = open(jsonfile, "r")-dataarray = []-datacount = 0- -for line in open(jsonfile):-line = f.readline()-lin = json.loads(line)-date = dateutil.parser.parse(lin["created_at"])-datacount += 1-dataarray.append(date.minute)- -minuteswithcount = [(x, dataarray.count(x)) for x in set(dataarray)]-f.close()-return minuteswithcount- - -def draw_date(files):-x = []-y = []-mwcs = parse_data(files)-for mwc in mwcs:-x.append(mwc[0])-y.append(mwc[1])- -plt.figure(figsize=(8,4))-plt.plot(x, y,label = files)-plt.legend()-plt.show()- -draw_date("data/2014-01-01-0.json")#!/usr/bin/env python +# -*- coding: utf-8 -*- + +import json +import dateutil.parser +import numpy as np +import matplotlib.mlab as mlab +import matplotlib.pyplot as plt + + +def parse_data(jsonfile): + f = open(jsonfile, "r") + dataarray = [] + datacount = 0 + + for line in open(jsonfile): + line = f.readline() + lin = json.loads(line) + date = dateutil.parser.parse(lin["created_at"]) + datacount += 1 + dataarray.append(date.minute) + + minuteswithcount = [(x, dataarray.count(x)) for x in set(dataarray)] + f.close() + return minuteswithcount + + +def draw_date(files): + x = [] + y = [] + mwcs = parse_data(files) + for mwc in mwcs: + x.append(mwc[0]) + y.append(mwc[1]) + + plt.figure(figsize=(8,4)) + plt.plot(x, y,label = files) + plt.legend() + plt.show() + +draw_date("data/2014-01-01-0.json")每周分析
继上篇之后，我们就可以分析用户的每周提交情况，以得出用户的真正的工具效率，每个程序员的工作时间可能是不一样的，如
@@ -2120,7 +2145,7 @@ Set up your git name and email, this is important so that your commits can be id 这是我的每周情况，显然如果把星期六移到前面的话，随着工作时间的增长，在github上的使用在下降，作为一个
a fulltime hacker who works best in the evening (around 8 pm).不过这个是osrc的分析结果。
-###python github 每周情况分析
+python github 每周情况分析
看一张分析后的结果
Feb Results @@ -2152,159 +2177,159 @@ Set up your git name and email, this is important so that your commits can be id
-6570, 7420, 11274, 12073, 12160, 12378, 12897, 8474, 7984, 12933, 13504, 13763, 13544, 12940, 7119, 7346, 13412, 14008, 12555###Python 数据分析
+Python 数据分析
重写了一个新的方法用于计算提交数，直至后面才意识到其实我们可以算行数就够了，但是方法上有点hack
-
+def get_minutes_counts_with_id(jsonfile):-datacount, dataarray = handle_json(jsonfile)-minuteswithcount = [(x, dataarray.count(x)) for x in set(dataarray)]-return minuteswithcount- - -def handle_json(jsonfile):-f = open(jsonfile, "r")-dataarray = []-datacount = 0- -for line in open(jsonfile):-line = f.readline()-lin = json.loads(line)-date = dateutil.parser.parse(lin["created_at"])-datacount += 1-dataarray.append(date.minute)- -f.close()-return datacount, dataarray- - -def get_minutes_count_num(jsonfile):-datacount, dataarray = handle_json(jsonfile)-return datacount- - -def get_month_total():-"""- -:rtype : object-"""-monthdaycount = []-for i in range(1, 20):-if i < 10:-filename = 'data/2014-02-0' + i.__str__() + '-0.json'-else:-filename = 'data/2014-02-' + i.__str__() + '-0.json'-monthdaycount.append(get_minutes_count_num(filename))-return monthdaycountdef get_minutes_counts_with_id(jsonfile): + datacount, dataarray = handle_json(jsonfile) + minuteswithcount = [(x, dataarray.count(x)) for x in set(dataarray)] + return minuteswithcount + + +def handle_json(jsonfile): + f = open(jsonfile, "r") + dataarray = [] + datacount = 0 + + for line in open(jsonfile): + line = f.readline() + lin = json.loads(line) + date = dateutil.parser.parse(lin["created_at"]) + datacount += 1 + dataarray.append(date.minute) + + f.close() + return datacount, dataarray + + +def get_minutes_count_num(jsonfile): + datacount, dataarray = handle_json(jsonfile) + return datacount + + +def get_month_total(): + """ + + :rtype : object + """ + monthdaycount = [] + for i in range(1, 20): + if i < 10: + filename = 'data/2014-02-0' + i.__str__() + '-0.json' + else: + filename = 'data/2014-02-' + i.__str__() + '-0.json' + monthdaycount.append(get_minutes_count_num(filename)) + return monthdaycount接着我们需要去遍历每个结果，后面的后面会发现这个效率真的是太低了，为什么木有多线程？
-###Python Matplotlib图表
+Python Matplotlib图表
让我们的matplotlib来做这些图表的工作
-
+if __name__ == '__main__':-results = pd.get_month_total()-print results- -plt.figure(figsize=(8, 4))-plt.plot(results.__getslice__(0, 7), label="first week")-plt.plot(results.__getslice__(7, 14), label="second week")-plt.plot(results.__getslice__(14, 21), label="third week")-plt.legend()-plt.show()if __name__ == '__main__': + results = pd.get_month_total() + print results + + plt.figure(figsize=(8, 4)) + plt.plot(results.__getslice__(0, 7), label="first week") + plt.plot(results.__getslice__(7, 14), label="second week") + plt.plot(results.__getslice__(14, 21), label="third week") + plt.legend() + plt.show()蓝色的是第一周，绿色的是第二周，红色的是第三周就有了上面的结果。
我们还需要优化方法，以及多线程的支持。
让我们分析之前的程序，然后再想办法做出优化。网上看到一篇文章http://www.huyng.com/posts/python-performance-analysis/讲的就是分析这部分内容的。
存储到数据库中
SQLite3
我们创建了一个名为
-userdata.db的数据库文件，然后创建了一个表，里面有owner,language,eventtype,name url
+def init_db():-conn = sqlite3.connect('userdata.db')-c = conn.cursor()-c.execute('''CREATE TABLE userinfo (owner text, language text, eventtype text, name text, url text)''')def init_db(): + conn = sqlite3.connect('userdata.db') + c = conn.cursor() + c.execute('''CREATE TABLE userinfo (owner text, language text, eventtype text, name text, url text)''')接着我们就可以查询数据，这里从结果讲起。
-
+def get_count(username):-count = 0-userinfo = []-condition = 'select * from userinfo where owener = \'' + str(username) + '\''-for zero in c.execute(condition):-count += 1-userinfo.append(zero)- -return count, userinfodef get_count(username): + count = 0 + userinfo = [] + condition = 'select * from userinfo where owener = \'' + str(username) + '\'' + for zero in c.execute(condition): + count += 1 + userinfo.append(zero) + + return count, userinfo当我查询
-gmszone的时候，也就是我自己就会有如下的结果
+(u'gmszone', u'ForkEvent', u'RESUME', u'TeX', u'https://github.com/gmszone/RESUME')-(u'gmszone', u'WatchEvent', u'iot-dashboard', u'JavaScript', u'https://github.com/gmszone/iot-dashboard')-(u'gmszone', u'PushEvent', u'wechat-wordpress', u'Ruby', u'https://github.com/gmszone/wechat-wordpress')-(u'gmszone', u'WatchEvent', u'iot', u'JavaScript', u'https://github.com/gmszone/iot')-(u'gmszone', u'CreateEvent', u'iot-doc', u'None', u'https://github.com/gmszone/iot-doc')-(u'gmszone', u'CreateEvent', u'iot-doc', u'None', u'https://github.com/gmszone/iot-doc')-(u'gmszone', u'PushEvent', u'iot-doc', u'TeX', u'https://github.com/gmszone/iot-doc')-(u'gmszone', u'PushEvent', u'iot-doc', u'TeX', u'https://github.com/gmszone/iot-doc')-(u'gmszone', u'PushEvent', u'iot-doc', u'TeX', u'https://github.com/gmszone/iot-doc')-109(u'gmszone', u'ForkEvent', u'RESUME', u'TeX', u'https://github.com/gmszone/RESUME') +(u'gmszone', u'WatchEvent', u'iot-dashboard', u'JavaScript', u'https://github.com/gmszone/iot-dashboard') +(u'gmszone', u'PushEvent', u'wechat-wordpress', u'Ruby', u'https://github.com/gmszone/wechat-wordpress') +(u'gmszone', u'WatchEvent', u'iot', u'JavaScript', u'https://github.com/gmszone/iot') +(u'gmszone', u'CreateEvent', u'iot-doc', u'None', u'https://github.com/gmszone/iot-doc') +(u'gmszone', u'CreateEvent', u'iot-doc', u'None', u'https://github.com/gmszone/iot-doc') +(u'gmszone', u'PushEvent', u'iot-doc', u'TeX', u'https://github.com/gmszone/iot-doc') +(u'gmszone', u'PushEvent', u'iot-doc', u'TeX', u'https://github.com/gmszone/iot-doc') +(u'gmszone', u'PushEvent', u'iot-doc', u'TeX', u'https://github.com/gmszone/iot-doc') +109一共有109个事件，有
Watch,Create,Push,Fork还有其他的， 项目主要有iot,RESUME,iot-dashboard,wechat-wordpress, 接着就是语言了，Tex,Javascript,Ruby,接着就是项目的url了。值得注意的是。
-
+-rw-r--r-- 1 fdhuang staff 905M Apr 12 14:59 userdata.db这个数据库文件有905M，不过查询结果相当让人满意，至少相对于原来的结果来说。
Python自带了对SQLite3的支持，然而我们还需要安装SQLite3
-
+brew install sqlite3或者是
-
+sudo port install sqlite3或者是Ubuntu的
-
+sudo apt-get install sqlite3openSUSE自然就是
-
+sudo zypper install sqlite3不过，用yast2也很不错，不是么。。
-###数据导入
+数据导入
需要注意的是这里是需要python2.7，起源于对gzip的上下文管理器的支持问题
-
+def handle_gzip_file(filename):-userinfo = []-with gzip.GzipFile(filename) as f:-events = [line.decode("utf-8", errors="ignore") for line in f]- -for n, line in enumerate(events):-try:-event = json.loads(line)-except:- -continue- -actor = event["actor"]-attrs = event.get("actor_attributes", {})-if actor is None or attrs.get("type") != "User":-continue- -key = actor.lower()- -repo = event.get("repository", {})-info = str(repo.get("owner")), str(repo.get("language")), str(event["type"]), str(repo.get("name")), str(-repo.get("url"))-userinfo.append(info)- -return userinfo- -def build_db_with_gzip():-init_db()-conn = sqlite3.connect('userdata.db')-c = conn.cursor()- -year = 2014-month = 3- -for day in range(1,31):-date_re = re.compile(r"([0-9]{4})-([0-9]{2})-([0-9]{2})-([0-9]+)\.json.gz")- -fn_template = os.path.join("march",-"{year}-{month:02d}-{day:02d}-{n}.json.gz")-kwargs = {"year": year, "month": month, "day": day, "n": "*"}-filenames = glob.glob(fn_template.format(**kwargs))- -for filename in filenames:-c.executemany('INSERT INTO userinfo VALUES (?,?,?,?,?)', handle_gzip_file(filename))- -conn.commit()-c.close()def handle_gzip_file(filename): + userinfo = [] + with gzip.GzipFile(filename) as f: + events = [line.decode("utf-8", errors="ignore") for line in f] + + for n, line in enumerate(events): + try: + event = json.loads(line) + except: + + continue + + actor = event["actor"] + attrs = event.get("actor_attributes", {}) + if actor is None or attrs.get("type") != "User": + continue + + key = actor.lower() + + repo = event.get("repository", {}) + info = str(repo.get("owner")), str(repo.get("language")), str(event["type"]), str(repo.get("name")), str( + repo.get("url")) + userinfo.append(info) + + return userinfo + +def build_db_with_gzip(): + init_db() + conn = sqlite3.connect('userdata.db') + c = conn.cursor() + + year = 2014 + month = 3 + + for day in range(1,31): + date_re = re.compile(r"([0-9]{4})-([0-9]{2})-([0-9]{2})-([0-9]+)\.json.gz") + + fn_template = os.path.join("march", + "{year}-{month:02d}-{day:02d}-{n}.json.gz") + kwargs = {"year": year, "month": month, "day": day, "n": "*"} + filenames = glob.glob(fn_template.format(**kwargs)) + + for filename in filenames: + c.executemany('INSERT INTO userinfo VALUES (?,?,?,?,?)', handle_gzip_file(filename)) + + conn.commit() + c.close()executemany可以插入多条数据，对于我们的数据来说，一小时的文件大概有五六千个会符合我们上面的安装，也就是有actor又有type才是我们需要记录的数据，我们只需要统计用户的那些事件，而非全部的事件。我们需要去遍历文件，然后找到合适的部分，这里只是要找
2014-03-01到2014-03-31的全部事件，而光这些数据的gz文件就有1.26G，同上面那些解压为json文件显得不合适，只能用遍历来处理。这里参考了osrc项目中的写法，或者说直接复制过来。
首先是正规匹配
-
+date_re = re.compile(r"([0-9]{4})-([0-9]{2})-([0-9]{2})-([0-9]+)\.json.gz")不过主要的还是在于
glob.globglob是python自己带的一个文件操作相关模块，用它可以查找符合自己目的的文件，就类似于Windows下的文件搜索，支持通配符操作。
@@ -2313,24 +2338,24 @@ Set up your git name and email, this is important so that your commits can be id最后代码可以见
更好的方案？
-###Redis
+Redis
查询用户事件总数
-
+import redis-r = redis.StrictRedis(host='localhost', port=6379, db=0)-pipe = pipe = r.pipeline()-pipe.zscore('osrc:user',"gmszone")-pipe.execute()import redis +r = redis.StrictRedis(host='localhost', port=6379, db=0) +pipe = pipe = r.pipeline() +pipe.zscore('osrc:user',"gmszone") +pipe.execute()系统返回了
-227.0,试试别人。
+>>> pipe.zscore('osrc:user',"dfm")-<redis.client.StrictPipeline object at 0x104fa7f50>->>> pipe.execute()-[425.0]->>>>>> pipe.zscore('osrc:user',"dfm") +<redis.client.StrictPipeline object at 0x104fa7f50> +>>> pipe.execute() +[425.0] +>>>看看主要是在哪一天提交的
-
+>>> pipe.hgetall('osrc:user:gmszone:day')-<redis.client.StrictPipeline object at 0x104fa7f50>->>> pipe.execute()-[{'1': '51', '0': '41', '3': '17', '2': '34', '5': '28', '4': '22', '6': '34'}]>>> pipe.hgetall('osrc:user:gmszone:day') +<redis.client.StrictPipeline object at 0x104fa7f50> +>>> pipe.execute() +[{'1': '51', '0': '41', '3': '17', '2': '34', '5': '28', '4': '22', '6': '34'}]结果大致如下图所示:
SMTWTFS @@ -2346,36 +2371,36 @@ Set up your git name and email, this is important so that your commits can be id蓝色的就是push事件，黄色的是create等等。
到这里我们算是知道了OSRC的数据库部分是如何工作的。
-####Redis 查询
+Redis 查询
主要代码如下所示
-
+def get_vector(user, pipe=None):- -r = redis.StrictRedis(host='localhost', port=6379, db=0)-no_pipe = False-if pipe is None:-pipe = pipe = r.pipeline()-no_pipe = True- -user = user.lower()-pipe.zscore(get_format("user"), user)-pipe.hgetall(get_format("user:{0}:day".format(user)))-pipe.zrevrange(get_format("user:{0}:event".format(user)), 0, -1,-withscores=True)-pipe.zcard(get_format("user:{0}:contribution".format(user)))-pipe.zcard(get_format("user:{0}:connection".format(user)))-pipe.zcard(get_format("user:{0}:repo".format(user)))-pipe.zcard(get_format("user:{0}:lang".format(user)))-pipe.zrevrange(get_format("user:{0}:lang".format(user)), 0, -1,-withscores=True)- -if no_pipe:-return pipe.execute()def get_vector(user, pipe=None): + + r = redis.StrictRedis(host='localhost', port=6379, db=0) + no_pipe = False + if pipe is None: + pipe = pipe = r.pipeline() + no_pipe = True + + user = user.lower() + pipe.zscore(get_format("user"), user) + pipe.hgetall(get_format("user:{0}:day".format(user))) + pipe.zrevrange(get_format("user:{0}:event".format(user)), 0, -1, + withscores=True) + pipe.zcard(get_format("user:{0}:contribution".format(user))) + pipe.zcard(get_format("user:{0}:connection".format(user))) + pipe.zcard(get_format("user:{0}:repo".format(user))) + pipe.zcard(get_format("user:{0}:lang".format(user))) + pipe.zrevrange(get_format("user:{0}:lang".format(user)), 0, -1, + withscores=True) + + if no_pipe: + return pipe.execute()结果在上一篇中显示出来了，也就是
[227.0, {'1': '51', '0': '41', '3': '17', '2': '34', '5': '28', '4': '22', '6': '34'}, [('PushEvent', 154.0), ('CreateEvent', 41.0), ('WatchEvent', 18.0), ('GollumEvent', 8.0), ('MemberEvent', 3.0), ('ForkEvent', 2.0), ('ReleaseEvent', 1.0)], 0, 0, 0, 11, [('CSS', 74.0), ('JavaScript', 60.0), ('Ruby', 12.0), ('TeX', 6.0), ('Python', 6.0), ('Java', 5.0), ('C++', 5.0), ('Assembly', 5.0), ('C', 3.0), ('Emacs Lisp', 2.0), ('Arduino', 2.0)]]有意思的是在这里生成了和自己相近的人
['alesdokshanin', 'hjiawei', 'andrewreedy', 'christj6', '1995eaton']osrc最有意思的一部分莫过于flann，当然说的也是系统后台的设计的一个很关键及有意思的部分。
-##邻近算法与相似用户
+邻近算法与相似用户
邻近算法是在这个分析过程中一个很有意思的东西。
邻近算法，或者说K最近邻(kNN，k-NearestNeighbor)分类算法可以说是整个数据挖掘分类技术中最简单的方法了。所谓K最近邻，就是k个最近的邻居的意思，说的是每个样本都可以用她最接近的k个邻居来代表。
@@ -2401,47 +2426,47 @@ Set up your git name and email, this is important so that your commits can be idosrc中用于解析的代码
-
+def parse_vector(results):-points = np.zeros(nvector)-total = int(results[0])- -points[0] = 1.0 / (total + 1)- -# Week means.-for k, v in results[1].iteritems():-points[1 + int(k)] = float(v) / total- -# Event types.-n = 8-for k, v in results[2]:-points[n + evttypes.index(k)] = float(v) / total- -# Number of contributions, connections and languages.-n += nevts-points[n] = 1.0 / (float(results[3]) + 1)-points[n + 1] = 1.0 / (float(results[4]) + 1)-points[n + 2] = 1.0 / (float(results[5]) + 1)-points[n + 3] = 1.0 / (float(results[6]) + 1)- -# Top languages.-n += 4-for k, v in results[7]:-if k in langs:-points[n + langs.index(k)] = float(v) / total-else:-# Unknown language.-points[-1] = float(v) / total- -return pointsdef parse_vector(results): + points = np.zeros(nvector) + total = int(results[0]) + + points[0] = 1.0 / (total + 1) + + # Week means. + for k, v in results[1].iteritems(): + points[1 + int(k)] = float(v) / total + + # Event types. + n = 8 + for k, v in results[2]: + points[n + evttypes.index(k)] = float(v) / total + + # Number of contributions, connections and languages. + n += nevts + points[n] = 1.0 / (float(results[3]) + 1) + points[n + 1] = 1.0 / (float(results[4]) + 1) + points[n + 2] = 1.0 / (float(results[5]) + 1) + points[n + 3] = 1.0 / (float(results[6]) + 1) + + # Top languages. + n += 4 + for k, v in results[7]: + if k in langs: + points[n + langs.index(k)] = float(v) / total + else: + # Unknown language. + points[-1] = float(v) / total + + return points这样也就返回我们需要的点数，然后我们可以用
-get_points来获取这些
+def get_points(usernames):-r = redis.StrictRedis(host='localhost', port=6379, db=0)-pipe = r.pipeline()- -results = get_vector(usernames)-points = np.zeros([len(usernames), nvector])-points = parse_vector(results)-return pointsdef get_points(usernames): + r = redis.StrictRedis(host='localhost', port=6379, db=0) + pipe = r.pipeline() + + results = get_vector(usernames) + points = np.zeros([len(usernames), nvector]) + points = parse_vector(results) + return points就会得到我们的相应的数据，接着找找和自己邻近的，看看结果。
[ 0.01298701 0.19736842 0. 0.30263158 0.21052632 0.19736842 0. 0.09210526 0. 0.22368421 0.01315789 0. 0. @@ -2510,97 +2535,97 @@ Set up your git name and email, this is important so that your commits can be id但是显然，他们都太重了。事实上，对于一个库来说，80%的人只需要其中20%的代码。于是，找到了https://github.com/stackp/promisejs，看了看用法，这就是我们需要的功能:
-
+function late(n) {-var p = new promise.Promise();-setTimeout(function() {-p.done(null, n);-}, n);-return p;-}- -late(100).then(-function(err, n) {-return late(n + 200);-}-).then(-function(err, n) {-return late(n + 300);-}-).then(-function(err, n) {-return late(n + 400);-}-).then(-function(err, n) {-alert(n);-}-);function late(n) { + var p = new promise.Promise(); + setTimeout(function() { + p.done(null, n); + }, n); + return p; +} + +late(100).then( + function(err, n) { + return late(n + 200); + } +).then( + function(err, n) { + return late(n + 300); + } +).then( + function(err, n) { + return late(n + 400); + } +).then( + function(err, n) { + alert(n); + } +);接着打开看看Promise对象，有我们需要的功能，但是又有一些功能超出我的需求。接着把自己不需要的需求去掉，这里函数最后就变成了
-
+function Promise() {-this._callbacks = [];-}- -Promise.prototype.then = function(func, context) {-var p;-if (this._isdone) {-p = func.apply(context, this.result);-} else {-p = new Promise();-this._callbacks.push(function () {-var res = func.apply(context, arguments);-if (res && typeof res.then === 'function') {-res.then(p.done, p);-}-});-}-return p;-};- -Promise.prototype.done = function() {-this.result = arguments;-this._isdone = true;-for (var i = 0; i < this._callbacks.length; i++) {-this._callbacks[i].apply(null, arguments);-}-this._callbacks = [];-};- -var promise = {-Promise: Promise-};function Promise() { + this._callbacks = []; +} + +Promise.prototype.then = function(func, context) { + var p; + if (this._isdone) { + p = func.apply(context, this.result); + } else { + p = new Promise(); + this._callbacks.push(function () { + var res = func.apply(context, arguments); + if (res && typeof res.then === 'function') { + res.then(p.done, p); + } + }); + } + return p; +}; + +Promise.prototype.done = function() { + this.result = arguments; + this._isdone = true; + for (var i = 0; i < this._callbacks.length; i++) { + this._callbacks[i].apply(null, arguments); + } + this._callbacks = []; +}; + +var promise = { + Promise: Promise +};需要注意的是:
License，不同的软件有不同的License，如MIT、GPL等等。最好能在遵循协议的情况下，使用别人的代码。实现第二个需求
由于已经有了现有的很多库，所以就可以直接参照（抄）别人写的代码。
-
+Lettuce.get = function (url, callback) {-Lettuce.send(url, 'GET', callback);-};- -Lettuce.load = function (url, callback) {-Lettuce.send(url, 'GET', callback);-};- -Lettuce.post = function (url, data, callback) {-Lettuce.send(url, 'POST', callback, data);-};- -Lettuce.send = function (url, method, callback, data) {-data = data || null;-var request = new XMLHttpRequest();-if (callback instanceof Function) {-request.onreadystatechange = function () {-if (request.readyState === 4 && (request.status === 200 || request.status === 0)) {-callback(request.responseText);-}-};-}-request.open(method, url, true);-if (data instanceof Object) {-data = JSON.stringify(data);-request.setRequestHeader('Content-Type', 'application/json');-}-request.setRequestHeader('X-Requested-With', 'XMLHttpRequest');-request.send(data);-};Lettuce.get = function (url, callback) { + Lettuce.send(url, 'GET', callback); +}; + +Lettuce.load = function (url, callback) { + Lettuce.send(url, 'GET', callback); +}; + +Lettuce.post = function (url, data, callback) { + Lettuce.send(url, 'POST', callback, data); +}; + +Lettuce.send = function (url, method, callback, data) { + data = data || null; + var request = new XMLHttpRequest(); + if (callback instanceof Function) { + request.onreadystatechange = function () { + if (request.readyState === 4 && (request.status === 200 || request.status === 0)) { + callback(request.responseText); + } + }; + } + request.open(method, url, true); + if (data instanceof Object) { + data = JSON.stringify(data); + request.setRequestHeader('Content-Type', 'application/json'); + } + request.setRequestHeader('X-Requested-With', 'XMLHttpRequest'); + request.send(data); +};如何以“正确的姿势”阅读开源软件代码
所有让你直接看最新源码的文章都是在扯淡，你应该从“某个版本”开始阅读代码。
@@ -2790,7 +2815,7 @@ Set up your git name and email, this is important so that your commits can be id代码: https://github.com/phodal/gmap-solr
技能树
这个可以从两部分说起:
-####重构Skill Tree
+重构 Skill Tree
原来的是
Skill Tree 代码: https://github.com/phodal/skillock
-####技能树Sherlock
+技能树Sherlock
而领域本身也是相似的，这可以解释为什么互联网公司都喜欢互相挖人，而一般都不会去华为、中兴等非互联网领域挖人。出了这个领域，你可能连个毕业生都不如。领域、业务同技术一样是不断强化知识的一个过程。Ritchie先实现了BCPL语言，而后设计了C语言，而BCPL语言一开始是基于CPL语言。
领域本身也在不断进化。
这也是下一个值得提高的地方。
-###其他
+其他
是时候写这个小结了。从不会写代码，到写代码是从0到1的过程，但是要从1到60都不是一件容易的事。无论是刷GitHub也好(不要是自动提交)，或者是换工作也好，我们都在不断地练习。
而练习是要分成不同的几个步骤，不仅仅局限于技术：